ConcurrentRebalancersTest.TwoConcurrentRebalancers/0: /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
  0
  num_crashes
    Which is: 1
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
  Actual: it does.
I20240502 21:35:08.509219 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 9023
I20240502 21:35:08.517789  9148 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:35:08.991585 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 9023
W20240502 21:35:09.023371  8946 catalog_manager.cc:4573] TS 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet cdf72d25f0f847e5998acbc53979e64d: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:35:09.060143 11027 connection.cc:535] server connection from 127.27.196.65:42255 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:09.060473 11295 connection.cc:535] server connection from 127.27.196.65:48577 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:09.060705 11431 connection.cc:535] server connection from 127.27.196.65:35315 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:09.060976  9172 connection.cc:535] client connection to 127.27.196.65:42177 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:35:09.061129 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 9157
W20240502 21:35:09.061138  9444 connection.cc:535] client connection to 127.27.196.65:42177 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:09.061269  9444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 14 similar messages]
W20240502 21:35:09.071081  9172 connection.cc:535] server connection from 127.27.196.65:48383 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:09.075014  9444 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Couldn't send request to peer 5543ddab5b484642b139b7c848932a58. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:35:09.075191  9444 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Couldn't send request to peer 5543ddab5b484642b139b7c848932a58. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:35:09.075279  9444 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Couldn't send request to peer 5543ddab5b484642b139b7c848932a58. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:35:09.075619  9172 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Couldn't send request to peer 5543ddab5b484642b139b7c848932a58. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:09.080994  9282 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:35:09.142300  9310 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Couldn't send request to peer 5543ddab5b484642b139b7c848932a58. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:35:09.254354 11429 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Couldn't send request to peer 5543ddab5b484642b139b7c848932a58. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:09.506129 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 9157
W20240502 21:35:09.576705 11431 connection.cc:535] server connection from 127.27.196.66:44945 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:09.577435  9313 connection.cc:535] client connection to 127.27.196.66:39071 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:35:09.577636 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 9293
W20240502 21:35:09.577643  9444 connection.cc:535] server connection from 127.27.196.66:58097 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:09.578287  9447 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Couldn't send request to peer 525a093ea5bf40dc9bbf9dc39895d670. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:35:09.579393  9313 consensus_peers.cc:485] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Couldn't send request to peer 525a093ea5bf40dc9bbf9dc39895d670. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:35:09.579629 11432 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240502 21:35:09.580541 11432 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Couldn't send request to peer 525a093ea5bf40dc9bbf9dc39895d670. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:09.585151  9419 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:35:09.632140 11298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111)
W20240502 21:35:09.634056 11298 consensus_peers.cc:485] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 -> Peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Couldn't send request to peer 525a093ea5bf40dc9bbf9dc39895d670. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:09.646793 24439 raft_consensus.cc:482] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:09.647092 24439 raft_consensus.cc:504] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:09.648105 24439 leader_election.cc:288] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:35:09.650122 11295 leader_election.cc:334] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:09.655943 11093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9240163cfac446d2a14dbd9fcbbf9ff7" candidate_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" is_pre_election: true
I20240502 21:35:09.656720 11296 leader_election.cc:302] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cbb5b06a6a44eb6b2145cd538b01af2; no voters: 5543ddab5b484642b139b7c848932a58, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:35:09.657083 24439 raft_consensus.cc:2726] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:35:09.762050 24840 raft_consensus.cc:482] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:09.762313 24840 raft_consensus.cc:504] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:35:09.763295 24840 leader_election.cc:288] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857)
I20240502 21:35:09.764715 11365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c306d724954a16be21a34aa4cc0b5f" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" is_pre_election: true
W20240502 21:35:09.764971  9444 leader_election.cc:334] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:09.765874  9444 leader_election.cc:302] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6; no voters: 4cbb5b06a6a44eb6b2145cd538b01af2, 5543ddab5b484642b139b7c848932a58
I20240502 21:35:09.766228 24840 raft_consensus.cc:2726] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:35:09.810922 11028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111)
W20240502 21:35:09.813287 11028 consensus_peers.cc:485] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 -> Peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Couldn't send request to peer 525a093ea5bf40dc9bbf9dc39895d670. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:09.842016 24840 raft_consensus.cc:482] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:09.842258 24840 raft_consensus.cc:504] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:09.844141 11365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b4486f03fd9474aa0bb2cb4c9149437" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" is_pre_election: true
I20240502 21:35:09.844358 11093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b4486f03fd9474aa0bb2cb4c9149437" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" is_pre_election: true
I20240502 21:35:09.844851 24840 leader_election.cc:288] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:35:09.844813  9444 leader_election.cc:334] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:09.845083  9445 leader_election.cc:302] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6; no voters: 5543ddab5b484642b139b7c848932a58, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:35:09.845468 24840 raft_consensus.cc:2726] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:35:09.861451 24439 raft_consensus.cc:482] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:09.861711 24439 raft_consensus.cc:504] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:35:09.862659 24439 leader_election.cc:288] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
I20240502 21:35:09.863636  9508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c306d724954a16be21a34aa4cc0b5f" candidate_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6" is_pre_election: true
I20240502 21:35:09.863916  9508 raft_consensus.cc:2454] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cbb5b06a6a44eb6b2145cd538b01af2 in term 1.
W20240502 21:35:09.864389 11295 leader_election.cc:334] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:09.864643 11295 leader_election.cc:302] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 4cbb5b06a6a44eb6b2145cd538b01af2; no voters: 5543ddab5b484642b139b7c848932a58
I20240502 21:35:09.864997 24439 raft_consensus.cc:2781] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:35:09.865100 24439 raft_consensus.cc:482] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:09.865199 24439 raft_consensus.cc:3037] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:35:09.867956 24439 raft_consensus.cc:504] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:35:09.868731 24439 leader_election.cc:288] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 election: Requested vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
I20240502 21:35:09.869786  9508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c306d724954a16be21a34aa4cc0b5f" candidate_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6"
I20240502 21:35:09.870394  9508 raft_consensus.cc:3037] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:35:09.870678 11295 leader_election.cc:334] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:09.874258  9508 raft_consensus.cc:2454] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cbb5b06a6a44eb6b2145cd538b01af2 in term 2.
I20240502 21:35:09.874787 11295 leader_election.cc:302] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 4cbb5b06a6a44eb6b2145cd538b01af2; no voters: 5543ddab5b484642b139b7c848932a58
I20240502 21:35:09.875701 24439 raft_consensus.cc:2781] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:35:09.875901 24439 raft_consensus.cc:686] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 LEADER]: Becoming Leader. State: Replica: 4cbb5b06a6a44eb6b2145cd538b01af2, State: Running, Role: LEADER
I20240502 21:35:09.876375 24439 consensus_queue.cc:237] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:35:09.884810  8955 catalog_manager.cc:5484] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 reported cstate change: term changed from 1 to 2, leader changed from 5543ddab5b484642b139b7c848932a58 (127.27.196.65) to 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71). New cstate: current_term: 2 leader_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } }
I20240502 21:35:09.886523 24840 raft_consensus.cc:482] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:09.886775 24840 raft_consensus.cc:504] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:35:09.887691 24840 leader_election.cc:288] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
W20240502 21:35:09.889883  9444 leader_election.cc:334] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:09.933768 24960 raft_consensus.cc:482] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:09.933990 24960 raft_consensus.cc:504] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:35:09.934746 24960 leader_election.cc:288] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
W20240502 21:35:09.936226 11298 leader_election.cc:334] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111)
W20240502 21:35:09.936391 11295 leader_election.cc:334] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:09.936504 11295 leader_election.cc:302] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cbb5b06a6a44eb6b2145cd538b01af2; no voters: 525a093ea5bf40dc9bbf9dc39895d670, 5543ddab5b484642b139b7c848932a58
I20240502 21:35:09.936774 24960 raft_consensus.cc:2726] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:35:10.011369 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 9293
W20240502 21:35:10.082175 11298 connection.cc:535] server connection from 127.27.196.67:43725 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:10.082196 11432 connection.cc:535] server connection from 127.27.196.67:39199 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:10.082201 11026 connection.cc:535] server connection from 127.27.196.67:54351 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:10.082702  9446 connection.cc:535] client connection to 127.27.196.67:43203 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:10.082769 11431 connection.cc:535] client connection to 127.27.196.67:43203 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:35:10.083046 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 9429
W20240502 21:35:10.084250 11431 consensus_peers.cc:485] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Couldn't send request to peer f2fdd596a8b44bbc848fe40f9f07fbe0. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.67:43203: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:35:10.084228  9446 leader_election.cc:334] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Network error: Client connection negotiation failed: client connection to 127.27.196.67:43203: connect: Connection refused (error 111)
I20240502 21:35:10.084426  9446 leader_election.cc:302] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6; no voters: 5543ddab5b484642b139b7c848932a58, f2fdd596a8b44bbc848fe40f9f07fbe0
I20240502 21:35:10.084792 24840 raft_consensus.cc:2726] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:35:10.084934  9446 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Couldn't send request to peer f2fdd596a8b44bbc848fe40f9f07fbe0. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.67:43203: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:35:10.085076  9446 consensus_peers.cc:485] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Couldn't send request to peer f2fdd596a8b44bbc848fe40f9f07fbe0. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.67:43203: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:10.090308  9554 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:35:10.160821 24539 raft_consensus.cc:482] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 525a093ea5bf40dc9bbf9dc39895d670)
I20240502 21:35:10.161068 24539 raft_consensus.cc:504] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:35:10.162042 24539 leader_election.cc:288] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
I20240502 21:35:10.162941 11090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51936fb1f678458a8795f8246590addb" candidate_uuid: "e0de0545fa0547d3929111e990e6d1b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" is_pre_election: true
W20240502 21:35:10.164436 11432 leader_election.cc:334] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111)
I20240502 21:35:10.164623 11432 leader_election.cc:302] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 525a093ea5bf40dc9bbf9dc39895d670, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:35:10.164920 24539 raft_consensus.cc:2726] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:35:10.402683 11295 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 -> Peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Couldn't send request to peer 5543ddab5b484642b139b7c848932a58. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:10.482177 24960 raft_consensus.cc:482] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:10.482432 24960 raft_consensus.cc:504] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:10.483389 24960 leader_election.cc:288] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
I20240502 21:35:10.484033 11090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b4486f03fd9474aa0bb2cb4c9149437" candidate_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" is_pre_election: true
W20240502 21:35:10.485229 11295 leader_election.cc:334] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:10.485353 11295 leader_election.cc:302] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4cbb5b06a6a44eb6b2145cd538b01af2; no voters: 5543ddab5b484642b139b7c848932a58, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:35:10.485646 24960 raft_consensus.cc:2726] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:35:10.490793 22998 raft_consensus.cc:482] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:10.491149 22998 raft_consensus.cc:504] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:10.492874 24539 raft_consensus.cc:482] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:10.493093 24539 raft_consensus.cc:504] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:10.493206 22998 leader_election.cc:288] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467)
I20240502 21:35:10.493376 22998 raft_consensus.cc:482] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:10.493870 24539 leader_election.cc:288] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
I20240502 21:35:10.494027 24539 raft_consensus.cc:482] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:10.494166 24539 raft_consensus.cc:504] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:10.494948 24539 leader_election.cc:288] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:35:10.498492 11431 leader_election.cc:334] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Network error: Client connection negotiation failed: client connection to 127.27.196.67:43203: connect: Connection refused (error 111)
W20240502 21:35:10.499827 11027 leader_election.cc:334] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Network error: Client connection negotiation failed: client connection to 127.27.196.67:43203: connect: Connection refused (error 111)
W20240502 21:35:10.503964 11429 leader_election.cc:334] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
W20240502 21:35:10.504124 11429 leader_election.cc:334] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:10.504221 11429 leader_election.cc:302] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 5543ddab5b484642b139b7c848932a58, f2fdd596a8b44bbc848fe40f9f07fbe0
I20240502 21:35:10.504525 24974 raft_consensus.cc:2726] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:35:10.505589  9508 raft_consensus.cc:1261] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Refusing update from remote peer 4cbb5b06a6a44eb6b2145cd538b01af2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 21:35:10.506424  9507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b4486f03fd9474aa0bb2cb4c9149437" candidate_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6" is_pre_election: true
I20240502 21:35:10.506721  9507 raft_consensus.cc:2454] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cbb5b06a6a44eb6b2145cd538b01af2 in term 1.
I20240502 21:35:10.507485 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 9429
I20240502 21:35:10.508826 22998 raft_consensus.cc:504] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:10.510265 22998 leader_election.cc:288] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857)
I20240502 21:35:10.510499 22998 raft_consensus.cc:482] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:10.510681 22998 raft_consensus.cc:504] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:10.511534 22998 leader_election.cc:288] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467)
I20240502 21:35:10.511972 22998 raft_consensus.cc:482] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:10.512230 22998 raft_consensus.cc:504] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:10.513098 22998 leader_election.cc:288] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:35:10.513913 24977 raft_consensus.cc:482] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:10.514159 24977 raft_consensus.cc:504] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:10.514500 11493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dedaa327b6479e8325f91481c1688e" candidate_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "e0de0545fa0547d3929111e990e6d1b2" is_pre_election: true
I20240502 21:35:10.514559 11495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1a31513bb64578a85adf1f622e5bc0" candidate_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: false dest_uuid: "e0de0545fa0547d3929111e990e6d1b2" is_pre_election: true
I20240502 21:35:10.514806 11493 raft_consensus.cc:2454] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d10e9e156ba74a97a221c85b68ab8bc4 in term 1.
I20240502 21:35:10.514873 11495 raft_consensus.cc:2454] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d10e9e156ba74a97a221c85b68ab8bc4 in term 3.
I20240502 21:35:10.515022 24978 raft_consensus.cc:482] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 525a093ea5bf40dc9bbf9dc39895d670)
I20240502 21:35:10.515195 11363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b4486f03fd9474aa0bb2cb4c9149437" candidate_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" is_pre_election: true
I20240502 21:35:10.515198 24978 raft_consensus.cc:504] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:35:10.515455 11363 raft_consensus.cc:2454] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d10e9e156ba74a97a221c85b68ab8bc4 in term 1.
I20240502 21:35:10.516165 24977 leader_election.cc:288] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857)
I20240502 21:35:10.516266 11090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1a31513bb64578a85adf1f622e5bc0" candidate_uuid: "e0de0545fa0547d3929111e990e6d1b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: false dest_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" is_pre_election: true
I20240502 21:35:10.516175 24978 leader_election.cc:288] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:35:10.516511 11090 raft_consensus.cc:2454] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0de0545fa0547d3929111e990e6d1b2 in term 3.
I20240502 21:35:10.517030 11090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dedaa327b6479e8325f91481c1688e" candidate_uuid: "e0de0545fa0547d3929111e990e6d1b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" is_pre_election: true
I20240502 21:35:10.517235 11090 raft_consensus.cc:2454] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0de0545fa0547d3929111e990e6d1b2 in term 1.
I20240502 21:35:10.517726 11363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9240163cfac446d2a14dbd9fcbbf9ff7" candidate_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" is_pre_election: true
W20240502 21:35:10.517827 11025 leader_election.cc:334] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
W20240502 21:35:10.517992 11025 leader_election.cc:334] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:10.517980 11363 raft_consensus.cc:2454] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d10e9e156ba74a97a221c85b68ab8bc4 in term 1.
I20240502 21:35:10.518106 11025 leader_election.cc:302] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d10e9e156ba74a97a221c85b68ab8bc4; no voters: 5543ddab5b484642b139b7c848932a58, f2fdd596a8b44bbc848fe40f9f07fbe0
I20240502 21:35:10.518499 11430 leader_election.cc:302] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d10e9e156ba74a97a221c85b68ab8bc4, e0de0545fa0547d3929111e990e6d1b2; no voters: 5543ddab5b484642b139b7c848932a58
W20240502 21:35:10.518616 11025 leader_election.cc:334] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:10.518654 24978 raft_consensus.cc:2726] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:35:10.518775 11025 leader_election.cc:334] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:10.519235 11025 leader_election.cc:302] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cbb5b06a6a44eb6b2145cd538b01af2, d10e9e156ba74a97a221c85b68ab8bc4; no voters: 
I20240502 21:35:10.519470 24974 raft_consensus.cc:2781] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 21:35:10.519526 24978 raft_consensus.cc:2781] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:35:10.519591 24974 raft_consensus.cc:482] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:10.519614 24978 raft_consensus.cc:482] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:10.519686 24974 raft_consensus.cc:3037] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:35:10.519728 24978 raft_consensus.cc:3037] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:35:10.522567 24974 raft_consensus.cc:504] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:10.523880 24974 leader_election.cc:288] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 election: Requested vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
I20240502 21:35:10.523751 24978 raft_consensus.cc:504] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:10.524443 11028 leader_election.cc:302] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d10e9e156ba74a97a221c85b68ab8bc4, e0de0545fa0547d3929111e990e6d1b2; no voters: 5543ddab5b484642b139b7c848932a58
I20240502 21:35:10.524616 24978 leader_election.cc:288] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 election: Requested vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857)
I20240502 21:35:10.524930 24978 raft_consensus.cc:2781] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240502 21:35:10.525058 11028 leader_election.cc:334] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111)
I20240502 21:35:10.525314 11028 leader_election.cc:302] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d10e9e156ba74a97a221c85b68ab8bc4; no voters: 525a093ea5bf40dc9bbf9dc39895d670, 5543ddab5b484642b139b7c848932a58
I20240502 21:35:10.525949 11363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9240163cfac446d2a14dbd9fcbbf9ff7" candidate_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2"
I20240502 21:35:10.526206 11363 raft_consensus.cc:3037] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:35:10.526608 11025 leader_election.cc:334] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
W20240502 21:35:10.526891 11025 leader_election.cc:334] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
W20240502 21:35:10.527724 11025 leader_election.cc:334] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:10.528062 24977 raft_consensus.cc:2726] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:35:10.528162 11429 leader_election.cc:334] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:10.525095 24978 raft_consensus.cc:482] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:10.528527 24978 raft_consensus.cc:3037] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:35:10.529630 11090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1a31513bb64578a85adf1f622e5bc0" candidate_uuid: "e0de0545fa0547d3929111e990e6d1b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: false dest_uuid: "d10e9e156ba74a97a221c85b68ab8bc4"
I20240502 21:35:10.530279 11363 raft_consensus.cc:2454] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d10e9e156ba74a97a221c85b68ab8bc4 in term 2.
I20240502 21:35:10.530879 11025 leader_election.cc:302] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cbb5b06a6a44eb6b2145cd538b01af2, d10e9e156ba74a97a221c85b68ab8bc4; no voters: 5543ddab5b484642b139b7c848932a58
I20240502 21:35:10.531334 24977 raft_consensus.cc:2781] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:35:10.531517 24977 raft_consensus.cc:686] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 LEADER]: Becoming Leader. State: Replica: d10e9e156ba74a97a221c85b68ab8bc4, State: Running, Role: LEADER
I20240502 21:35:10.531893 24977 consensus_queue.cc:237] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
W20240502 21:35:10.532020 11028 leader_election.cc:334] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111)
I20240502 21:35:10.532395 11028 leader_election.cc:302] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d10e9e156ba74a97a221c85b68ab8bc4; no voters: 525a093ea5bf40dc9bbf9dc39895d670, 5543ddab5b484642b139b7c848932a58
I20240502 21:35:10.532575 24978 raft_consensus.cc:504] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:10.532902 24976 raft_consensus.cc:2726] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:35:10.533103 11090 raft_consensus.cc:2379] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e0de0545fa0547d3929111e990e6d1b2 in current term 4: Already voted for candidate d10e9e156ba74a97a221c85b68ab8bc4 in this term.
I20240502 21:35:10.534294 11430 leader_election.cc:302] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 5543ddab5b484642b139b7c848932a58, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:35:10.535576 24978 leader_election.cc:288] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 4 election: Requested vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467)
I20240502 21:35:10.535831 11493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1a31513bb64578a85adf1f622e5bc0" candidate_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: false dest_uuid: "e0de0545fa0547d3929111e990e6d1b2"
I20240502 21:35:10.536392 11493 raft_consensus.cc:2379] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d10e9e156ba74a97a221c85b68ab8bc4 in current term 4: Already voted for candidate e0de0545fa0547d3929111e990e6d1b2 in this term.
W20240502 21:35:10.536592 11025 leader_election.cc:334] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:10.536522  8955 catalog_manager.cc:5484] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 reported cstate change: term changed from 1 to 2, leader changed from 5543ddab5b484642b139b7c848932a58 (127.27.196.65) to d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69). New cstate: current_term: 2 leader_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:35:10.537031 24974 raft_consensus.cc:2726] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:35:10.537170 11028 leader_election.cc:302] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d10e9e156ba74a97a221c85b68ab8bc4; no voters: 5543ddab5b484642b139b7c848932a58, e0de0545fa0547d3929111e990e6d1b2
I20240502 21:35:10.537658 24977 raft_consensus.cc:2726] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240502 21:35:10.593297 11430 connection.cc:535] server connection from 127.27.196.68:36505 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:10.594033 11025 connection.cc:535] client connection to 127.27.196.68:40931 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:35:10.594092 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 11009
W20240502 21:35:10.595700 11295 leader_election.cc:334] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): Network error: Client connection negotiation failed: client connection to 127.27.196.68:40931: connect: Connection refused (error 111)
W20240502 21:35:10.596032 11025 leader_election.cc:334] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): Network error: Client connection negotiation failed: client connection to 127.27.196.68:40931: connect: Connection refused (error 111)
I20240502 21:35:10.596201 11025 leader_election.cc:302] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4cbb5b06a6a44eb6b2145cd538b01af2, d10e9e156ba74a97a221c85b68ab8bc4; no voters: 19f162e164854f2da623a4bb8f9e6bf6, 5543ddab5b484642b139b7c848932a58
W20240502 21:35:10.596174 11295 consensus_peers.cc:485] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 -> Peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): Couldn't send request to peer 19f162e164854f2da623a4bb8f9e6bf6. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.68:40931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:35:10.596415 11295 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 -> Peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): Couldn't send request to peer 19f162e164854f2da623a4bb8f9e6bf6. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.68:40931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:10.596693 24977 raft_consensus.cc:2726] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:35:10.607625 11134 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:35:10.617102 24960 raft_consensus.cc:482] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f2fdd596a8b44bbc848fe40f9f07fbe0)
I20240502 21:35:10.617380 24960 raft_consensus.cc:504] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:35:10.618643 24960 leader_election.cc:288] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467)
W20240502 21:35:10.620044 11297 leader_election.cc:334] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Network error: Client connection negotiation failed: client connection to 127.27.196.67:43203: connect: Connection refused (error 111)
W20240502 21:35:10.625006 11298 leader_election.cc:334] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111)
I20240502 21:35:10.625309 11298 leader_election.cc:302] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4cbb5b06a6a44eb6b2145cd538b01af2; no voters: 525a093ea5bf40dc9bbf9dc39895d670, f2fdd596a8b44bbc848fe40f9f07fbe0
I20240502 21:35:10.625679 24960 raft_consensus.cc:2726] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:35:10.631333 11493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ee481a3ece47a0905e93c66dbf949d" candidate_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "e0de0545fa0547d3929111e990e6d1b2" is_pre_election: true
I20240502 21:35:11.005424 24960 raft_consensus.cc:482] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 525a093ea5bf40dc9bbf9dc39895d670)
I20240502 21:35:11.005681 24960 raft_consensus.cc:504] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:35:11.006465 24960 leader_election.cc:288] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:35:11.008548 24974 raft_consensus.cc:482] T 58c7004ceb044f5fa9dec54910046b69 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 525a093ea5bf40dc9bbf9dc39895d670)
W20240502 21:35:11.008594 11298 leader_election.cc:334] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111)
W20240502 21:35:11.008729 11297 leader_election.cc:334] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Network error: Client connection negotiation failed: client connection to 127.27.196.67:43203: connect: Connection refused (error 111)
I20240502 21:35:11.008828 11297 leader_election.cc:302] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cbb5b06a6a44eb6b2145cd538b01af2; no voters: 525a093ea5bf40dc9bbf9dc39895d670, f2fdd596a8b44bbc848fe40f9f07fbe0
I20240502 21:35:11.008785 24974 raft_consensus.cc:504] T 58c7004ceb044f5fa9dec54910046b69 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:35:11.009128 24960 raft_consensus.cc:2726] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:35:11.009742 24974 leader_election.cc:288] T 58c7004ceb044f5fa9dec54910046b69 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
W20240502 21:35:11.011180 11432 leader_election.cc:334] T 58c7004ceb044f5fa9dec54910046b69 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111)
W20240502 21:35:11.011356 11429 leader_election.cc:334] T 58c7004ceb044f5fa9dec54910046b69 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): Network error: Client connection negotiation failed: client connection to 127.27.196.68:40931: connect: Connection refused (error 111)
I20240502 21:35:11.011467 11429 leader_election.cc:302] T 58c7004ceb044f5fa9dec54910046b69 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 19f162e164854f2da623a4bb8f9e6bf6, 525a093ea5bf40dc9bbf9dc39895d670
I20240502 21:35:11.011749 24974 raft_consensus.cc:2726] T 58c7004ceb044f5fa9dec54910046b69 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:35:11.027899 24976 raft_consensus.cc:482] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 525a093ea5bf40dc9bbf9dc39895d670)
I20240502 21:35:11.029347 24976 raft_consensus.cc:504] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:35:11.030251 24976 leader_election.cc:288] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467)
I20240502 21:35:11.030601 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 11009
I20240502 21:35:11.056970 24993 raft_consensus.cc:482] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f2fdd596a8b44bbc848fe40f9f07fbe0)
I20240502 21:35:11.057287 24993 raft_consensus.cc:504] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:35:11.058303 24993 leader_election.cc:288] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:35:11.059873 11297 leader_election.cc:334] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Network error: Client connection negotiation failed: client connection to 127.27.196.67:43203: connect: Connection refused (error 111)
W20240502 21:35:11.100962 11296 connection.cc:535] client connection to 127.27.196.69:42049 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:11.101176 11295 connection.cc:535] server connection from 127.27.196.69:42039 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:35:11.101531 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 11280
W20240502 21:35:11.101539 11429 connection.cc:535] server connection from 127.27.196.69:43471 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:11.102533 11296 leader_election.cc:334] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Network error: Client connection negotiation failed: client connection to 127.27.196.69:42049: connect: Connection refused (error 111)
I20240502 21:35:11.102712 11296 leader_election.cc:302] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cbb5b06a6a44eb6b2145cd538b01af2; no voters: d10e9e156ba74a97a221c85b68ab8bc4, f2fdd596a8b44bbc848fe40f9f07fbe0
I20240502 21:35:11.103055 24993 raft_consensus.cc:2726] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:35:11.108706 11404 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:35:11.109686  8946 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 106 similar messages]
W20240502 21:35:11.111377  8946 catalog_manager.cc:4573] TS 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet cdf72d25f0f847e5998acbc53979e64d: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:35:11.130290 11430 consensus_peers.cc:485] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Couldn't send request to peer d10e9e156ba74a97a221c85b68ab8bc4. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.69:42049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:11.183755 24974 raft_consensus.cc:482] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 19f162e164854f2da623a4bb8f9e6bf6)
I20240502 21:35:11.184011 24974 raft_consensus.cc:504] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:35:11.184863 24974 leader_election.cc:288] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857)
W20240502 21:35:11.187721 11429 leader_election.cc:334] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): Network error: Client connection negotiation failed: client connection to 127.27.196.68:40931: connect: Connection refused (error 111)
W20240502 21:35:11.511669 11429 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Couldn't send request to peer 5543ddab5b484642b139b7c848932a58. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:35:11.514046 24974 raft_consensus.cc:482] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f2fdd596a8b44bbc848fe40f9f07fbe0)
I20240502 21:35:11.514271 24974 raft_consensus.cc:504] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:35:11.514649 24999 raft_consensus.cc:482] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f2fdd596a8b44bbc848fe40f9f07fbe0)
I20240502 21:35:11.514856 24999 raft_consensus.cc:504] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:35:11.515312 24974 leader_election.cc:288] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:35:11.515551 24974 raft_consensus.cc:482] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f2fdd596a8b44bbc848fe40f9f07fbe0)
I20240502 21:35:11.519084 24960 raft_consensus.cc:482] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f2fdd596a8b44bbc848fe40f9f07fbe0)
I20240502 21:35:11.519347 24960 raft_consensus.cc:504] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:35:11.520263 24960 leader_election.cc:288] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467)
W20240502 21:35:11.524487 11431 leader_election.cc:334] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Network error: Client connection negotiation failed: client connection to 127.27.196.67:43203: connect: Connection refused (error 111)
I20240502 21:35:11.525964 24999 leader_election.cc:288] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:35:11.515759 24974 raft_consensus.cc:504] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
W20240502 21:35:11.526609 11431 leader_election.cc:334] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Network error: Client connection negotiation failed: client connection to 127.27.196.67:43203: connect: Connection refused (error 111)
I20240502 21:35:11.527364 24974 leader_election.cc:288] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857)
I20240502 21:35:11.528257 25001 raft_consensus.cc:482] T 1497b781f36c43b39d22a05bb524c59d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 19f162e164854f2da623a4bb8f9e6bf6)
W20240502 21:35:11.528399 11431 leader_election.cc:334] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Network error: Client connection negotiation failed: client connection to 127.27.196.67:43203: connect: Connection refused (error 111)
I20240502 21:35:11.528463 25001 raft_consensus.cc:504] T 1497b781f36c43b39d22a05bb524c59d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:35:11.529785 25001 leader_election.cc:288] T 1497b781f36c43b39d22a05bb524c59d P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467)
W20240502 21:35:11.530066 11429 leader_election.cc:334] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:11.530263 11429 leader_election.cc:302] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 5543ddab5b484642b139b7c848932a58, f2fdd596a8b44bbc848fe40f9f07fbe0
W20240502 21:35:11.530995 11295 leader_election.cc:334] T 1497b781f36c43b39d22a05bb524c59d P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): Network error: Client connection negotiation failed: client connection to 127.27.196.68:40931: connect: Connection refused (error 111)
I20240502 21:35:11.531458 24974 raft_consensus.cc:482] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f2fdd596a8b44bbc848fe40f9f07fbe0)
I20240502 21:35:11.531788 25000 raft_consensus.cc:2726] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:35:11.531651 24974 raft_consensus.cc:504] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:35:11.531858 11365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ee481a3ece47a0905e93c66dbf949d" candidate_uuid: "e0de0545fa0547d3929111e990e6d1b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" is_pre_election: true
I20240502 21:35:11.531811 11363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1497b781f36c43b39d22a05bb524c59d" candidate_uuid: "e0de0545fa0547d3929111e990e6d1b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" is_pre_election: true
I20240502 21:35:11.532277 11365 raft_consensus.cc:2454] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0de0545fa0547d3929111e990e6d1b2 in term 1.
I20240502 21:35:11.532414 24974 leader_election.cc:288] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857)
I20240502 21:35:11.532656 11363 raft_consensus.cc:2454] T 1497b781f36c43b39d22a05bb524c59d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0de0545fa0547d3929111e990e6d1b2 in term 1.
W20240502 21:35:11.533126 11297 leader_election.cc:334] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Network error: Client connection negotiation failed: client connection to 127.27.196.67:43203: connect: Connection refused (error 111)
I20240502 21:35:11.533771 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 11280
I20240502 21:35:11.535987 11493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f35aff2b104c01a9d9d48a7e9c9b23" candidate_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "e0de0545fa0547d3929111e990e6d1b2" is_pre_election: true
I20240502 21:35:11.536273 11493 raft_consensus.cc:2454] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cbb5b06a6a44eb6b2145cd538b01af2 in term 2.
I20240502 21:35:11.536326 11429 leader_election.cc:302] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cbb5b06a6a44eb6b2145cd538b01af2, e0de0545fa0547d3929111e990e6d1b2; no voters: 19f162e164854f2da623a4bb8f9e6bf6
I20240502 21:35:11.536854 24974 raft_consensus.cc:2781] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:35:11.536971 24974 raft_consensus.cc:482] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 19f162e164854f2da623a4bb8f9e6bf6)
I20240502 21:35:11.537084 24974 raft_consensus.cc:3037] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:35:11.537089 11432 leader_election.cc:334] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111)
I20240502 21:35:11.537621 11432 leader_election.cc:302] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 525a093ea5bf40dc9bbf9dc39895d670, f2fdd596a8b44bbc848fe40f9f07fbe0
W20240502 21:35:11.537750 11431 leader_election.cc:334] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Network error: Client connection negotiation failed: client connection to 127.27.196.67:43203: connect: Connection refused (error 111)
I20240502 21:35:11.538005 25000 raft_consensus.cc:2726] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:11.538229 11432 leader_election.cc:334] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111)
I20240502 21:35:11.538398 11432 leader_election.cc:302] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4cbb5b06a6a44eb6b2145cd538b01af2, e0de0545fa0547d3929111e990e6d1b2; no voters: 525a093ea5bf40dc9bbf9dc39895d670, f2fdd596a8b44bbc848fe40f9f07fbe0
I20240502 21:35:11.538893 25000 raft_consensus.cc:2726] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:35:11.540418 24974 raft_consensus.cc:504] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:35:11.541093 24974 leader_election.cc:288] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 election: Requested vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857)
W20240502 21:35:11.542403 11429 leader_election.cc:334] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): Network error: Client connection negotiation failed: client connection to 127.27.196.68:40931: connect: Connection refused (error 111)
W20240502 21:35:11.608669 11429 connection.cc:535] client connection to 127.27.196.71:35857 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:11.609081 11431 connection.cc:535] server connection from 127.27.196.71:57407 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:35:11.609077 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 11414
W20240502 21:35:11.610947 11429 leader_election.cc:334] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857): Network error: Client connection negotiation failed: client connection to 127.27.196.71:35857: connect: Connection refused (error 111)
I20240502 21:35:11.611183 11429 leader_election.cc:302] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 4cbb5b06a6a44eb6b2145cd538b01af2, f2fdd596a8b44bbc848fe40f9f07fbe0
W20240502 21:35:11.611696 11429 leader_election.cc:334] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857): Network error: Client connection negotiation failed: client connection to 127.27.196.71:35857: connect: Connection refused (error 111)
I20240502 21:35:11.611896 24974 raft_consensus.cc:2726] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 21:35:11.611864 11429 leader_election.cc:302] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 19f162e164854f2da623a4bb8f9e6bf6, 4cbb5b06a6a44eb6b2145cd538b01af2
I20240502 21:35:11.612249 25000 raft_consensus.cc:2726] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 21:35:11.616735 11536 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:35:12.019749 24999 raft_consensus.cc:482] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 525a093ea5bf40dc9bbf9dc39895d670)
I20240502 21:35:12.020713 25000 raft_consensus.cc:482] T a2f3edaf1c874cefb820f027f90d1185 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 19f162e164854f2da623a4bb8f9e6bf6)
I20240502 21:35:12.020906 25000 raft_consensus.cc:504] T a2f3edaf1c874cefb820f027f90d1185 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:35:12.022954 25000 leader_election.cc:288] T a2f3edaf1c874cefb820f027f90d1185 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857)
W20240502 21:35:12.022948 11432 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Couldn't send request to peer 525a093ea5bf40dc9bbf9dc39895d670. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:35:12.024171 24999 raft_consensus.cc:504] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:35:12.025067 24999 leader_election.cc:288] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
I20240502 21:35:12.027312 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 11414
I20240502 21:35:12.096514 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 8930
I20240502 21:35:12.103364  8991 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:35:12.490141 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 8930
2024-05-02T21:35:12Z chronyd exiting
I20240502 21:35:12.576421 28433 test_util.cc:175] -----------------------------------------------
I20240502 21:35:12.576527 28433 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0

Note: This is test shard 3 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest
[ RUN      ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 21:25:19.876695 28433 test_util.cc:268] Using random seed: 1031327265
I20240502 21:25:19.890856 28433 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:25:19.890967 28433 ts_itest-base.cc:113] --------------
I20240502 21:25:19.891028 28433 ts_itest-base.cc:114] 8 tablet servers
I20240502 21:25:19.891076 28433 ts_itest-base.cc:115] 3 replicas per TS
I20240502 21:25:19.891165 28433 ts_itest-base.cc:116] --------------
2024-05-02T21:25:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:25:19Z Disabled control of system clock
I20240502 21:25:19.924599 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.196.126:39011
--webserver_interface=127.27.196.126
--webserver_port=0
--builtin_ntp_servers=127.27.196.84:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.196.126:39011
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 21:25:20.346508 28445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:20.346931 28445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:20.347039 28445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:20.355401 28445 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:20.355557 28445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:25:20.355612 28445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:20.355664 28445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:25:20.355719 28445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:25:20.366351 28445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.27.196.126:39011
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.196.126:39011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.27.196.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:20.367983 28445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:20.370028 28445 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:20.383253 28450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:20.383392 28451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:20.384027 28453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:20.384470 28445 server_base.cc:1034] running on GCE node
I20240502 21:25:20.385439 28445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:20.387096 28445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:20.388394 28445 hybrid_clock.cc:648] HybridClock initialized: now 1714685120388315 us; error 62 us; skew 500 ppm
I20240502 21:25:20.388799 28445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:20.398890 28445 webserver.cc:460] Webserver started at http://127.27.196.126:41753/ using document root <none> and password file <none>
I20240502 21:25:20.400058 28445 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:20.400218 28445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:20.400676 28445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:20.404290 28445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "1432c9f353d6488784f05a5917e63d8c"
format_stamp: "Formatted at 2024-05-02 21:25:20 on dist-test-slave-3bs1"
I20240502 21:25:20.405254 28445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "1432c9f353d6488784f05a5917e63d8c"
format_stamp: "Formatted at 2024-05-02 21:25:20 on dist-test-slave-3bs1"
I20240502 21:25:20.412972 28445 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.006s	sys 0.002s
I20240502 21:25:20.417994 28459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:20.420058 28445 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.001s
I20240502 21:25:20.420325 28445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
uuid: "1432c9f353d6488784f05a5917e63d8c"
format_stamp: "Formatted at 2024-05-02 21:25:20 on dist-test-slave-3bs1"
I20240502 21:25:20.420596 28445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:20.439278 28445 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:25:20.440232 28445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:20.440542 28445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:20.464803 28445 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.126:39011
I20240502 21:25:20.464812 28511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.126:39011 every 8 connection(s)
I20240502 21:25:20.466380 28445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 21:25:20.470541 28512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:20.471480 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 28445
I20240502 21:25:20.471729 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:25:20.481787 28512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1432c9f353d6488784f05a5917e63d8c: Bootstrap starting.
I20240502 21:25:20.484999 28512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1432c9f353d6488784f05a5917e63d8c: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:20.486140 28512 log.cc:824] T 00000000000000000000000000000000 P 1432c9f353d6488784f05a5917e63d8c: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:20.489038 28512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1432c9f353d6488784f05a5917e63d8c: No bootstrap required, opened a new log
I20240502 21:25:20.497690 28512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1432c9f353d6488784f05a5917e63d8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1432c9f353d6488784f05a5917e63d8c" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 39011 } }
I20240502 21:25:20.498196 28512 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1432c9f353d6488784f05a5917e63d8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:20.498322 28512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1432c9f353d6488784f05a5917e63d8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1432c9f353d6488784f05a5917e63d8c, State: Initialized, Role: FOLLOWER
I20240502 21:25:20.499217 28512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1432c9f353d6488784f05a5917e63d8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1432c9f353d6488784f05a5917e63d8c" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 39011 } }
I20240502 21:25:20.499481 28512 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1432c9f353d6488784f05a5917e63d8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:20.499749 28512 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1432c9f353d6488784f05a5917e63d8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:20.499940 28512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1432c9f353d6488784f05a5917e63d8c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:20.502276 28512 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1432c9f353d6488784f05a5917e63d8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1432c9f353d6488784f05a5917e63d8c" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 39011 } }
I20240502 21:25:20.502817 28512 leader_election.cc:302] T 00000000000000000000000000000000 P 1432c9f353d6488784f05a5917e63d8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1432c9f353d6488784f05a5917e63d8c; no voters: 
I20240502 21:25:20.503684 28512 leader_election.cc:288] T 00000000000000000000000000000000 P 1432c9f353d6488784f05a5917e63d8c [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 21:25:20.503866 28517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1432c9f353d6488784f05a5917e63d8c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:20.504757 28517 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1432c9f353d6488784f05a5917e63d8c [term 1 LEADER]: Becoming Leader. State: Replica: 1432c9f353d6488784f05a5917e63d8c, State: Running, Role: LEADER
I20240502 21:25:20.505645 28517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1432c9f353d6488784f05a5917e63d8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1432c9f353d6488784f05a5917e63d8c" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 39011 } }
I20240502 21:25:20.506155 28512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1432c9f353d6488784f05a5917e63d8c [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:25:20.509747 28518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1432c9f353d6488784f05a5917e63d8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1432c9f353d6488784f05a5917e63d8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1432c9f353d6488784f05a5917e63d8c" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 39011 } } }
I20240502 21:25:20.510308 28518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1432c9f353d6488784f05a5917e63d8c [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:20.510241 28519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1432c9f353d6488784f05a5917e63d8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1432c9f353d6488784f05a5917e63d8c. Latest consensus state: current_term: 1 leader_uuid: "1432c9f353d6488784f05a5917e63d8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1432c9f353d6488784f05a5917e63d8c" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 39011 } } }
I20240502 21:25:20.510509 28519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1432c9f353d6488784f05a5917e63d8c [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:20.513080 28523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:25:20.521176 28523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:25:20.532649 28523 catalog_manager.cc:1347] Generated new cluster ID: d67dd981d9f3493a945b7808fdb2bad6
I20240502 21:25:20.532842 28523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:25:20.544163 28523 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:25:20.545912 28523 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:25:20.562958 28523 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1432c9f353d6488784f05a5917e63d8c: Generated new TSK 0
I20240502 21:25:20.564363 28523 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:25:20.583066 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.65:0
--local_ip_for_outbound_sockets=127.27.196.65
--webserver_interface=127.27.196.65
--webserver_port=0
--tserver_master_addrs=127.27.196.126:39011
--builtin_ntp_servers=127.27.196.84:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 21:25:20.995532 28536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:20.995877 28536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:20.996050 28536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:21.004770 28536 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:21.004942 28536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:21.005122 28536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.65
I20240502 21:25:21.015421 28536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.27.196.65
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.27.196.126:39011
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.65
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:21.016993 28536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:21.019100 28536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:21.037109 28542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:21.037169 28541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:21.038066 28544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:21.040472 28536 server_base.cc:1034] running on GCE node
I20240502 21:25:21.041527 28536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:21.043203 28536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:21.044492 28536 hybrid_clock.cc:648] HybridClock initialized: now 1714685121044389 us; error 95 us; skew 500 ppm
I20240502 21:25:21.044979 28536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:21.047907 28536 webserver.cc:460] Webserver started at http://127.27.196.65:36389/ using document root <none> and password file <none>
I20240502 21:25:21.048777 28536 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:21.048911 28536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:21.049279 28536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:21.052515 28536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "04e36797791842e489aee30de261c4c7"
format_stamp: "Formatted at 2024-05-02 21:25:21 on dist-test-slave-3bs1"
I20240502 21:25:21.053457 28536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "04e36797791842e489aee30de261c4c7"
format_stamp: "Formatted at 2024-05-02 21:25:21 on dist-test-slave-3bs1"
I20240502 21:25:21.060434 28536 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.003s	sys 0.005s
I20240502 21:25:21.066740 28550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:21.069111 28536 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.000s
I20240502 21:25:21.069360 28536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "04e36797791842e489aee30de261c4c7"
format_stamp: "Formatted at 2024-05-02 21:25:21 on dist-test-slave-3bs1"
I20240502 21:25:21.069671 28536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:21.090353 28536 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:25:21.091493 28536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:21.091840 28536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:21.093098 28536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:21.095026 28536 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:21.095160 28536 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:21.095281 28536 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:21.095369 28536 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:21.136793 28536 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.65:44983
I20240502 21:25:21.136849 28666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.65:44983 every 8 connection(s)
I20240502 21:25:21.138289 28536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:25:21.138942 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 28536
I20240502 21:25:21.139205 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:25:21.143886 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.66:0
--local_ip_for_outbound_sockets=127.27.196.66
--webserver_interface=127.27.196.66
--webserver_port=0
--tserver_master_addrs=127.27.196.126:39011
--builtin_ntp_servers=127.27.196.84:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 21:25:21.154795 28667 heartbeater.cc:347] Connected to a master server at 127.27.196.126:39011
I20240502 21:25:21.155221 28667 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:21.156288 28667 heartbeater.cc:511] Master 127.27.196.126:39011 requested a full tablet report, sending...
I20240502 21:25:21.159492 28476 ts_manager.cc:194] Registered new tserver with Master: 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983)
I20240502 21:25:21.161293 28476 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.65:40515
W20240502 21:25:21.575047 28671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:21.575507 28671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:21.575704 28671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:21.583792 28671 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:21.583938 28671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:21.584105 28671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.66
I20240502 21:25:21.593892 28671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.27.196.66
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.27.196.126:39011
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.66
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:21.595507 28671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:21.597465 28671 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:21.611922 28676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:21.612181 28677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:21.612949 28679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:21.613858 28671 server_base.cc:1034] running on GCE node
I20240502 21:25:21.614766 28671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:21.616230 28671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:21.617550 28671 hybrid_clock.cc:648] HybridClock initialized: now 1714685121617489 us; error 44 us; skew 500 ppm
I20240502 21:25:21.618049 28671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:21.620847 28671 webserver.cc:460] Webserver started at http://127.27.196.66:41507/ using document root <none> and password file <none>
I20240502 21:25:21.621735 28671 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:21.621850 28671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:21.622208 28671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:21.625336 28671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "8786a1419d3f4bc79b88c54ce0675aa6"
format_stamp: "Formatted at 2024-05-02 21:25:21 on dist-test-slave-3bs1"
I20240502 21:25:21.626276 28671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "8786a1419d3f4bc79b88c54ce0675aa6"
format_stamp: "Formatted at 2024-05-02 21:25:21 on dist-test-slave-3bs1"
I20240502 21:25:21.632978 28671 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.007s	sys 0.001s
I20240502 21:25:21.637110 28685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:21.639379 28671 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 21:25:21.639658 28671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "8786a1419d3f4bc79b88c54ce0675aa6"
format_stamp: "Formatted at 2024-05-02 21:25:21 on dist-test-slave-3bs1"
I20240502 21:25:21.639961 28671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:21.659189 28671 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:25:21.660409 28671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:21.660756 28671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:21.662118 28671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:21.664518 28671 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:21.664690 28671 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:21.664860 28671 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:21.664947 28671 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:21.703755 28671 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.66:41815
I20240502 21:25:21.703830 28802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.66:41815 every 8 connection(s)
I20240502 21:25:21.705157 28671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:25:21.712656 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 28671
I20240502 21:25:21.712929 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:25:21.717821 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.67:0
--local_ip_for_outbound_sockets=127.27.196.67
--webserver_interface=127.27.196.67
--webserver_port=0
--tserver_master_addrs=127.27.196.126:39011
--builtin_ntp_servers=127.27.196.84:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 21:25:21.722358 28803 heartbeater.cc:347] Connected to a master server at 127.27.196.126:39011
I20240502 21:25:21.722824 28803 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:21.724086 28803 heartbeater.cc:511] Master 127.27.196.126:39011 requested a full tablet report, sending...
I20240502 21:25:21.726338 28476 ts_manager.cc:194] Registered new tserver with Master: 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815)
I20240502 21:25:21.727586 28476 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.66:40763
W20240502 21:25:22.157591 28807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:22.157944 28807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:22.158064 28807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 21:25:22.165529 28667 heartbeater.cc:503] Master 127.27.196.126:39011 was elected leader, sending a full tablet report...
W20240502 21:25:22.170392 28807 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:22.170573 28807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:22.170821 28807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.67
I20240502 21:25:22.184991 28807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.27.196.67
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.27.196.126:39011
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.67
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:22.187148 28807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:22.190148 28807 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:22.209033 28813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:22.209748 28814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:22.210961 28816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:22.214267 28807 server_base.cc:1034] running on GCE node
I20240502 21:25:22.215356 28807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:22.216792 28807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:22.218174 28807 hybrid_clock.cc:648] HybridClock initialized: now 1714685122218078 us; error 44 us; skew 500 ppm
I20240502 21:25:22.218849 28807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:22.222172 28807 webserver.cc:460] Webserver started at http://127.27.196.67:39571/ using document root <none> and password file <none>
I20240502 21:25:22.223304 28807 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:22.223598 28807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:22.224138 28807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:22.228137 28807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8733381e64da4e91a8a54e23b01e8967"
format_stamp: "Formatted at 2024-05-02 21:25:22 on dist-test-slave-3bs1"
I20240502 21:25:22.229177 28807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8733381e64da4e91a8a54e23b01e8967"
format_stamp: "Formatted at 2024-05-02 21:25:22 on dist-test-slave-3bs1"
I20240502 21:25:22.236683 28807 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.003s	sys 0.005s
I20240502 21:25:22.241659 28822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:22.244170 28807 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.001s	sys 0.004s
I20240502 21:25:22.244467 28807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8733381e64da4e91a8a54e23b01e8967"
format_stamp: "Formatted at 2024-05-02 21:25:22 on dist-test-slave-3bs1"
I20240502 21:25:22.244800 28807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:22.294325 28807 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:25:22.295459 28807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:22.295789 28807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:22.297008 28807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:22.299225 28807 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:22.299338 28807 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:22.299468 28807 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:22.299541 28807 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:22.341634 28807 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.67:42545
I20240502 21:25:22.341698 28939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.67:42545 every 8 connection(s)
I20240502 21:25:22.343206 28807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:25:22.346343 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 28807
I20240502 21:25:22.346573 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:25:22.350492 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.68:0
--local_ip_for_outbound_sockets=127.27.196.68
--webserver_interface=127.27.196.68
--webserver_port=0
--tserver_master_addrs=127.27.196.126:39011
--builtin_ntp_servers=127.27.196.84:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 21:25:22.360977 28940 heartbeater.cc:347] Connected to a master server at 127.27.196.126:39011
I20240502 21:25:22.361419 28940 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:22.362666 28940 heartbeater.cc:511] Master 127.27.196.126:39011 requested a full tablet report, sending...
I20240502 21:25:22.364789 28476 ts_manager.cc:194] Registered new tserver with Master: 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545)
I20240502 21:25:22.365649 28476 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.67:47859
I20240502 21:25:22.730556 28803 heartbeater.cc:503] Master 127.27.196.126:39011 was elected leader, sending a full tablet report...
W20240502 21:25:22.778019 28944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:22.778491 28944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:22.778700 28944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:22.787978 28944 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:22.788189 28944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:22.788404 28944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.68
I20240502 21:25:22.799988 28944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.27.196.68
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.27.196.126:39011
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.68
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:22.801695 28944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:22.803826 28944 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:22.818403 28951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:22.818639 28950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:22.818933 28953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:22.819804 28944 server_base.cc:1034] running on GCE node
I20240502 21:25:22.820710 28944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:22.821872 28944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:22.823199 28944 hybrid_clock.cc:648] HybridClock initialized: now 1714685122823135 us; error 61 us; skew 500 ppm
I20240502 21:25:22.823590 28944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:22.826529 28944 webserver.cc:460] Webserver started at http://127.27.196.68:39951/ using document root <none> and password file <none>
I20240502 21:25:22.827503 28944 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:22.827641 28944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:22.828014 28944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:22.831215 28944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "52c5becf6262415cab0408b3968ed4c0"
format_stamp: "Formatted at 2024-05-02 21:25:22 on dist-test-slave-3bs1"
I20240502 21:25:22.832186 28944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "52c5becf6262415cab0408b3968ed4c0"
format_stamp: "Formatted at 2024-05-02 21:25:22 on dist-test-slave-3bs1"
I20240502 21:25:22.840092 28944 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.006s	sys 0.001s
I20240502 21:25:22.845775 28959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:22.847990 28944 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 21:25:22.848225 28944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "52c5becf6262415cab0408b3968ed4c0"
format_stamp: "Formatted at 2024-05-02 21:25:22 on dist-test-slave-3bs1"
I20240502 21:25:22.848486 28944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:22.886008 28944 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:25:22.887040 28944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:22.887382 28944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:22.888507 28944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:22.890239 28944 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:22.890349 28944 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:22.890482 28944 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:22.890560 28944 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:22.928727 28944 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.68:41795
I20240502 21:25:22.928793 29074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.68:41795 every 8 connection(s)
I20240502 21:25:22.930260 28944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:25:22.940553 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 28944
I20240502 21:25:22.940824 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 21:25:22.945416 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.69:0
--local_ip_for_outbound_sockets=127.27.196.69
--webserver_interface=127.27.196.69
--webserver_port=0
--tserver_master_addrs=127.27.196.126:39011
--builtin_ntp_servers=127.27.196.84:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 21:25:22.946461 29075 heartbeater.cc:347] Connected to a master server at 127.27.196.126:39011
I20240502 21:25:22.946802 29075 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:22.947809 29075 heartbeater.cc:511] Master 127.27.196.126:39011 requested a full tablet report, sending...
I20240502 21:25:22.949574 28476 ts_manager.cc:194] Registered new tserver with Master: 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795)
I20240502 21:25:22.950410 28476 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.68:43611
W20240502 21:25:23.349123 29079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:23.349498 29079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:23.349664 29079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:23.358696 29079 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:23.358865 29079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:23.359045 29079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.69
I20240502 21:25:23.368917 29079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.27.196.69
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.27.196.126:39011
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.69
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:23.369706 28940 heartbeater.cc:503] Master 127.27.196.126:39011 was elected leader, sending a full tablet report...
I20240502 21:25:23.370702 29079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:23.372892 29079 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:23.387910 29085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:23.387818 29084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:23.388135 29087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:23.389223 29079 server_base.cc:1034] running on GCE node
I20240502 21:25:23.390002 29079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:23.391263 29079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:23.392648 29079 hybrid_clock.cc:648] HybridClock initialized: now 1714685123392577 us; error 61 us; skew 500 ppm
I20240502 21:25:23.393038 29079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:23.396090 29079 webserver.cc:460] Webserver started at http://127.27.196.69:33279/ using document root <none> and password file <none>
I20240502 21:25:23.396977 29079 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:23.397125 29079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:23.397492 29079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:23.400840 29079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "a1e2dba8e1a3403295cb394f7090f849"
format_stamp: "Formatted at 2024-05-02 21:25:23 on dist-test-slave-3bs1"
I20240502 21:25:23.401808 29079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "a1e2dba8e1a3403295cb394f7090f849"
format_stamp: "Formatted at 2024-05-02 21:25:23 on dist-test-slave-3bs1"
I20240502 21:25:23.408463 29079 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.004s	sys 0.004s
I20240502 21:25:23.412761 29093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:23.415253 29079 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.006s	sys 0.000s
I20240502 21:25:23.415537 29079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "a1e2dba8e1a3403295cb394f7090f849"
format_stamp: "Formatted at 2024-05-02 21:25:23 on dist-test-slave-3bs1"
I20240502 21:25:23.415853 29079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:23.440348 29079 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:25:23.441586 29079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:23.441926 29079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:23.443353 29079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:23.445775 29079 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:23.445927 29079 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:23.446066 29079 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:23.446138 29079 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:23.485204 29079 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.69:41545
I20240502 21:25:23.485270 29209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.69:41545 every 8 connection(s)
I20240502 21:25:23.486789 29079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:25:23.491376 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 29079
I20240502 21:25:23.491654 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 21:25:23.496276 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.70:0
--local_ip_for_outbound_sockets=127.27.196.70
--webserver_interface=127.27.196.70
--webserver_port=0
--tserver_master_addrs=127.27.196.126:39011
--builtin_ntp_servers=127.27.196.84:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 21:25:23.504518 29210 heartbeater.cc:347] Connected to a master server at 127.27.196.126:39011
I20240502 21:25:23.505059 29210 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:23.506162 29210 heartbeater.cc:511] Master 127.27.196.126:39011 requested a full tablet report, sending...
I20240502 21:25:23.508291 28476 ts_manager.cc:194] Registered new tserver with Master: a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69:41545)
I20240502 21:25:23.509598 28476 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.69:34217
W20240502 21:25:23.880285 29214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:23.880620 29214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:23.880781 29214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:23.888592 29214 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:23.888736 29214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:23.888860 29214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.70
I20240502 21:25:23.898388 29214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.27.196.70
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.27.196.126:39011
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.70
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:23.899835 29214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:23.901783 29214 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:23.915138 29220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:23.915210 29219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:23.915848 29222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:23.916142 29214 server_base.cc:1034] running on GCE node
I20240502 21:25:23.916771 29214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:23.917786 29214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:23.919004 29214 hybrid_clock.cc:648] HybridClock initialized: now 1714685123918953 us; error 43 us; skew 500 ppm
I20240502 21:25:23.919405 29214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:23.922227 29214 webserver.cc:460] Webserver started at http://127.27.196.70:38885/ using document root <none> and password file <none>
I20240502 21:25:23.923043 29214 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:23.923197 29214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:23.923533 29214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:23.926285 29214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "fc5aa2dc1f7b4554906785d693e272d0"
format_stamp: "Formatted at 2024-05-02 21:25:23 on dist-test-slave-3bs1"
I20240502 21:25:23.927181 29214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "fc5aa2dc1f7b4554906785d693e272d0"
format_stamp: "Formatted at 2024-05-02 21:25:23 on dist-test-slave-3bs1"
I20240502 21:25:23.933439 29214 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.000s	sys 0.004s
I20240502 21:25:23.937644 29228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:23.939579 29214 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.000s
I20240502 21:25:23.939795 29214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "fc5aa2dc1f7b4554906785d693e272d0"
format_stamp: "Formatted at 2024-05-02 21:25:23 on dist-test-slave-3bs1"
I20240502 21:25:23.940037 29214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:23.953558 29075 heartbeater.cc:503] Master 127.27.196.126:39011 was elected leader, sending a full tablet report...
I20240502 21:25:23.956357 29214 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:25:23.957254 29214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:23.957540 29214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:23.958701 29214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:23.960338 29214 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:23.960444 29214 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:23.960567 29214 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:23.960644 29214 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:23.997892 29214 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.70:43969
I20240502 21:25:23.998023 29344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.70:43969 every 8 connection(s)
I20240502 21:25:23.999465 29214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 21:25:24.003609 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 29214
I20240502 21:25:24.003815 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 21:25:24.008410 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.71:0
--local_ip_for_outbound_sockets=127.27.196.71
--webserver_interface=127.27.196.71
--webserver_port=0
--tserver_master_addrs=127.27.196.126:39011
--builtin_ntp_servers=127.27.196.84:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 21:25:24.017467 29345 heartbeater.cc:347] Connected to a master server at 127.27.196.126:39011
I20240502 21:25:24.017822 29345 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:24.018815 29345 heartbeater.cc:511] Master 127.27.196.126:39011 requested a full tablet report, sending...
I20240502 21:25:24.020792 28476 ts_manager.cc:194] Registered new tserver with Master: fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70:43969)
I20240502 21:25:24.022221 28476 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.70:37553
W20240502 21:25:24.411989 29349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:24.412437 29349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:24.412556 29349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:24.421381 29349 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:24.421566 29349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:24.421728 29349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.71
I20240502 21:25:24.432868 29349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.27.196.71
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.27.196.126:39011
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.71
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:24.434793 29349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:24.437070 29349 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:24.452970 29354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:24.453410 29355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:24.453569 29357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:24.454918 29349 server_base.cc:1034] running on GCE node
I20240502 21:25:24.455765 29349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:24.457000 29349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:24.458241 29349 hybrid_clock.cc:648] HybridClock initialized: now 1714685124458176 us; error 64 us; skew 500 ppm
I20240502 21:25:24.458626 29349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:24.461586 29349 webserver.cc:460] Webserver started at http://127.27.196.71:44401/ using document root <none> and password file <none>
I20240502 21:25:24.462462 29349 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:24.462630 29349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:24.463029 29349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:24.466687 29349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "a9ca254f89984b3f9ed62020bd6a4d02"
format_stamp: "Formatted at 2024-05-02 21:25:24 on dist-test-slave-3bs1"
I20240502 21:25:24.467908 29349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "a9ca254f89984b3f9ed62020bd6a4d02"
format_stamp: "Formatted at 2024-05-02 21:25:24 on dist-test-slave-3bs1"
I20240502 21:25:24.476244 29349 fs_manager.cc:691] Time spent creating directory manager: real 0.008s	user 0.007s	sys 0.000s
I20240502 21:25:24.480465 29363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:24.482091 29349 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.001s
I20240502 21:25:24.482278 29349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "a9ca254f89984b3f9ed62020bd6a4d02"
format_stamp: "Formatted at 2024-05-02 21:25:24 on dist-test-slave-3bs1"
I20240502 21:25:24.482563 29349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:24.513121 29210 heartbeater.cc:503] Master 127.27.196.126:39011 was elected leader, sending a full tablet report...
I20240502 21:25:24.514668 29349 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:25:24.515810 29349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:24.516146 29349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:24.517273 29349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:24.519057 29349 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:24.519217 29349 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.001s
I20240502 21:25:24.519348 29349 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:24.519446 29349 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:24.558338 29349 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.71:36827
I20240502 21:25:24.558495 29478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.71:36827 every 8 connection(s)
I20240502 21:25:24.559980 29349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 21:25:24.564606 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 29349
I20240502 21:25:24.565111 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 21:25:24.569844 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.72:0
--local_ip_for_outbound_sockets=127.27.196.72
--webserver_interface=127.27.196.72
--webserver_port=0
--tserver_master_addrs=127.27.196.126:39011
--builtin_ntp_servers=127.27.196.84:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 21:25:24.579154 29479 heartbeater.cc:347] Connected to a master server at 127.27.196.126:39011
I20240502 21:25:24.579555 29479 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:24.580592 29479 heartbeater.cc:511] Master 127.27.196.126:39011 requested a full tablet report, sending...
I20240502 21:25:24.582321 28476 ts_manager.cc:194] Registered new tserver with Master: a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71:36827)
I20240502 21:25:24.583230 28476 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.71:45407
W20240502 21:25:24.952579 29483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:24.952978 29483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:24.953161 29483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:24.961788 29483 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:24.961957 29483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:24.962153 29483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.72
I20240502 21:25:24.974188 29483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.27.196.72
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.27.196.126:39011
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.72
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:24.976665 29483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:24.979434 29483 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:24.993543 29488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:24.994251 29489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:24.994544 29491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:24.994956 29483 server_base.cc:1034] running on GCE node
I20240502 21:25:24.995921 29483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:24.996977 29483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:24.998204 29483 hybrid_clock.cc:648] HybridClock initialized: now 1714685124998137 us; error 56 us; skew 500 ppm
I20240502 21:25:24.998544 29483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:25.001067 29483 webserver.cc:460] Webserver started at http://127.27.196.72:35791/ using document root <none> and password file <none>
I20240502 21:25:25.001811 29483 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:25.001962 29483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:25.002296 29483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:25.005016 29483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "4e8bd1e7c1f848b0bdd2570de97506d8"
format_stamp: "Formatted at 2024-05-02 21:25:24 on dist-test-slave-3bs1"
I20240502 21:25:25.005869 29483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "4e8bd1e7c1f848b0bdd2570de97506d8"
format_stamp: "Formatted at 2024-05-02 21:25:24 on dist-test-slave-3bs1"
I20240502 21:25:25.012605 29483 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.001s
I20240502 21:25:25.017844 29497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:25.020108 29483 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.002s	sys 0.001s
I20240502 21:25:25.020339 29483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "4e8bd1e7c1f848b0bdd2570de97506d8"
format_stamp: "Formatted at 2024-05-02 21:25:24 on dist-test-slave-3bs1"
I20240502 21:25:25.020591 29483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:25.025488 29345 heartbeater.cc:503] Master 127.27.196.126:39011 was elected leader, sending a full tablet report...
I20240502 21:25:25.050056 29483 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:25:25.050997 29483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:25.051334 29483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:25.052388 29483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:25.054026 29483 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:25.054119 29483 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:25.054234 29483 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:25.054293 29483 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:25.093165 29483 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.72:34997
I20240502 21:25:25.093225 29612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.72:34997 every 8 connection(s)
I20240502 21:25:25.094616 29483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 21:25:25.104815 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 29483
I20240502 21:25:25.105001 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 21:25:25.107834 29613 heartbeater.cc:347] Connected to a master server at 127.27.196.126:39011
I20240502 21:25:25.108150 29613 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:25.109273 29613 heartbeater.cc:511] Master 127.27.196.126:39011 requested a full tablet report, sending...
I20240502 21:25:25.110901 28476 ts_manager.cc:194] Registered new tserver with Master: 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72:34997)
I20240502 21:25:25.111760 28476 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.72:44629
I20240502 21:25:25.124974 28433 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 21:25:25.153079 28433 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 21:25:25.153296 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 29079
I20240502 21:25:25.163928 29205 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:25.248203 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 29079
I20240502 21:25:25.268105 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 29214
I20240502 21:25:25.280328 29340 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:25.368150 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 29214
I20240502 21:25:25.397368 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 29349
I20240502 21:25:25.412544 29474 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:25.501806 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 29349
I20240502 21:25:25.523448 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 29483
I20240502 21:25:25.533878 29608 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:25.625836 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 29483
I20240502 21:25:29.403544 28476 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49308:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 21:25:29.449551 28736 tablet_service.cc:1467] Processing CreateTablet for tablet 20322b184e2f4a88b1abb569e6916f7c (DEFAULT_TABLE table=rebalance_test_table_0 [id=e74fde389c564592a4c3ed6f66c55951]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:29.449551 29008 tablet_service.cc:1467] Processing CreateTablet for tablet d25783a912884d2ba72c586d6cd7dfdd (DEFAULT_TABLE table=rebalance_test_table_0 [id=e74fde389c564592a4c3ed6f66c55951]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:29.451328 28735 tablet_service.cc:1467] Processing CreateTablet for tablet d25783a912884d2ba72c586d6cd7dfdd (DEFAULT_TABLE table=rebalance_test_table_0 [id=e74fde389c564592a4c3ed6f66c55951]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:29.451294 29007 tablet_service.cc:1467] Processing CreateTablet for tablet e94280a60b6d4902b866ae3c30e40470 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e74fde389c564592a4c3ed6f66c55951]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:29.452244 28736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20322b184e2f4a88b1abb569e6916f7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:29.452237 29008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d25783a912884d2ba72c586d6cd7dfdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:29.452785 29007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e94280a60b6d4902b866ae3c30e40470. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:29.453076 28735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d25783a912884d2ba72c586d6cd7dfdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:29.459812 28597 tablet_service.cc:1467] Processing CreateTablet for tablet e94280a60b6d4902b866ae3c30e40470 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e74fde389c564592a4c3ed6f66c55951]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:29.461612 28873 tablet_service.cc:1467] Processing CreateTablet for tablet 20322b184e2f4a88b1abb569e6916f7c (DEFAULT_TABLE table=rebalance_test_table_0 [id=e74fde389c564592a4c3ed6f66c55951]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:29.462597 28597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e94280a60b6d4902b866ae3c30e40470. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:29.460749 28598 tablet_service.cc:1467] Processing CreateTablet for tablet d25783a912884d2ba72c586d6cd7dfdd (DEFAULT_TABLE table=rebalance_test_table_0 [id=e74fde389c564592a4c3ed6f66c55951]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:29.463490 28873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20322b184e2f4a88b1abb569e6916f7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:29.463511 28598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d25783a912884d2ba72c586d6cd7dfdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:29.459899 28871 tablet_service.cc:1467] Processing CreateTablet for tablet e94280a60b6d4902b866ae3c30e40470 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e74fde389c564592a4c3ed6f66c55951]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:29.467836 28871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e94280a60b6d4902b866ae3c30e40470. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:29.471189 28599 tablet_service.cc:1467] Processing CreateTablet for tablet 20322b184e2f4a88b1abb569e6916f7c (DEFAULT_TABLE table=rebalance_test_table_0 [id=e74fde389c564592a4c3ed6f66c55951]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:29.471889 28599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20322b184e2f4a88b1abb569e6916f7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:29.488277 29645 tablet_bootstrap.cc:492] T e94280a60b6d4902b866ae3c30e40470 P 8733381e64da4e91a8a54e23b01e8967: Bootstrap starting.
I20240502 21:25:29.489207 29643 tablet_bootstrap.cc:492] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0: Bootstrap starting.
I20240502 21:25:29.493311 29644 tablet_bootstrap.cc:492] T 20322b184e2f4a88b1abb569e6916f7c P 8786a1419d3f4bc79b88c54ce0675aa6: Bootstrap starting.
I20240502 21:25:29.496832 29643 tablet_bootstrap.cc:654] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:29.496832 29645 tablet_bootstrap.cc:654] T e94280a60b6d4902b866ae3c30e40470 P 8733381e64da4e91a8a54e23b01e8967: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:29.498302 29645 log.cc:824] T e94280a60b6d4902b866ae3c30e40470 P 8733381e64da4e91a8a54e23b01e8967: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:29.498301 29643 log.cc:824] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:29.498986 29644 tablet_bootstrap.cc:654] T 20322b184e2f4a88b1abb569e6916f7c P 8786a1419d3f4bc79b88c54ce0675aa6: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:29.500826 29644 log.cc:824] T 20322b184e2f4a88b1abb569e6916f7c P 8786a1419d3f4bc79b88c54ce0675aa6: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:29.512061 29644 tablet_bootstrap.cc:492] T 20322b184e2f4a88b1abb569e6916f7c P 8786a1419d3f4bc79b88c54ce0675aa6: No bootstrap required, opened a new log
I20240502 21:25:29.512454 29644 ts_tablet_manager.cc:1367] T 20322b184e2f4a88b1abb569e6916f7c P 8786a1419d3f4bc79b88c54ce0675aa6: Time spent bootstrapping tablet: real 0.019s	user 0.012s	sys 0.004s
I20240502 21:25:29.513752 29646 tablet_bootstrap.cc:492] T e94280a60b6d4902b866ae3c30e40470 P 04e36797791842e489aee30de261c4c7: Bootstrap starting.
I20240502 21:25:29.519740 29646 tablet_bootstrap.cc:654] T e94280a60b6d4902b866ae3c30e40470 P 04e36797791842e489aee30de261c4c7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:29.521770 29646 log.cc:824] T e94280a60b6d4902b866ae3c30e40470 P 04e36797791842e489aee30de261c4c7: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:29.523496 29644 raft_consensus.cc:348] T 20322b184e2f4a88b1abb569e6916f7c P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:25:29.524149 29644 raft_consensus.cc:374] T 20322b184e2f4a88b1abb569e6916f7c P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:29.524286 29644 raft_consensus.cc:729] T 20322b184e2f4a88b1abb569e6916f7c P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8786a1419d3f4bc79b88c54ce0675aa6, State: Initialized, Role: FOLLOWER
I20240502 21:25:29.525367 29644 consensus_queue.cc:260] T 20322b184e2f4a88b1abb569e6916f7c P 8786a1419d3f4bc79b88c54ce0675aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:25:29.526515 29643 tablet_bootstrap.cc:492] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0: No bootstrap required, opened a new log
I20240502 21:25:29.526571 29646 tablet_bootstrap.cc:492] T e94280a60b6d4902b866ae3c30e40470 P 04e36797791842e489aee30de261c4c7: No bootstrap required, opened a new log
I20240502 21:25:29.526571 29645 tablet_bootstrap.cc:492] T e94280a60b6d4902b866ae3c30e40470 P 8733381e64da4e91a8a54e23b01e8967: No bootstrap required, opened a new log
I20240502 21:25:29.526926 29646 ts_tablet_manager.cc:1367] T e94280a60b6d4902b866ae3c30e40470 P 04e36797791842e489aee30de261c4c7: Time spent bootstrapping tablet: real 0.013s	user 0.009s	sys 0.000s
I20240502 21:25:29.526945 29645 ts_tablet_manager.cc:1367] T e94280a60b6d4902b866ae3c30e40470 P 8733381e64da4e91a8a54e23b01e8967: Time spent bootstrapping tablet: real 0.039s	user 0.009s	sys 0.024s
I20240502 21:25:29.526978 29643 ts_tablet_manager.cc:1367] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0: Time spent bootstrapping tablet: real 0.038s	user 0.009s	sys 0.023s
I20240502 21:25:29.537012 29646 raft_consensus.cc:348] T e94280a60b6d4902b866ae3c30e40470 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:25:29.537014 29645 raft_consensus.cc:348] T e94280a60b6d4902b866ae3c30e40470 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:25:29.537595 29646 raft_consensus.cc:374] T e94280a60b6d4902b866ae3c30e40470 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:29.537717 29646 raft_consensus.cc:729] T e94280a60b6d4902b866ae3c30e40470 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e36797791842e489aee30de261c4c7, State: Initialized, Role: FOLLOWER
I20240502 21:25:29.537771 29645 raft_consensus.cc:374] T e94280a60b6d4902b866ae3c30e40470 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:29.537963 29645 raft_consensus.cc:729] T e94280a60b6d4902b866ae3c30e40470 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8733381e64da4e91a8a54e23b01e8967, State: Initialized, Role: FOLLOWER
I20240502 21:25:29.538877 29645 consensus_queue.cc:260] T e94280a60b6d4902b866ae3c30e40470 P 8733381e64da4e91a8a54e23b01e8967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:25:29.540539 29644 ts_tablet_manager.cc:1398] T 20322b184e2f4a88b1abb569e6916f7c P 8786a1419d3f4bc79b88c54ce0675aa6: Time spent starting tablet: real 0.028s	user 0.021s	sys 0.006s
I20240502 21:25:29.541720 29644 tablet_bootstrap.cc:492] T d25783a912884d2ba72c586d6cd7dfdd P 8786a1419d3f4bc79b88c54ce0675aa6: Bootstrap starting.
I20240502 21:25:29.541954 29646 consensus_queue.cc:260] T e94280a60b6d4902b866ae3c30e40470 P 04e36797791842e489aee30de261c4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:25:29.545076 29644 tablet_bootstrap.cc:654] T d25783a912884d2ba72c586d6cd7dfdd P 8786a1419d3f4bc79b88c54ce0675aa6: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:29.546895 29645 ts_tablet_manager.cc:1398] T e94280a60b6d4902b866ae3c30e40470 P 8733381e64da4e91a8a54e23b01e8967: Time spent starting tablet: real 0.020s	user 0.006s	sys 0.012s
I20240502 21:25:29.546469 29643 raft_consensus.cc:348] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:25:29.547269 29643 raft_consensus.cc:374] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:29.547420 29643 raft_consensus.cc:729] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c5becf6262415cab0408b3968ed4c0, State: Initialized, Role: FOLLOWER
I20240502 21:25:29.548616 29643 consensus_queue.cc:260] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:25:29.549775 29645 tablet_bootstrap.cc:492] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967: Bootstrap starting.
I20240502 21:25:29.550010 29646 ts_tablet_manager.cc:1398] T e94280a60b6d4902b866ae3c30e40470 P 04e36797791842e489aee30de261c4c7: Time spent starting tablet: real 0.023s	user 0.012s	sys 0.005s
I20240502 21:25:29.551915 29645 tablet_bootstrap.cc:654] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:29.556838 29646 tablet_bootstrap.cc:492] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7: Bootstrap starting.
I20240502 21:25:29.559281 29646 tablet_bootstrap.cc:654] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:29.562162 29646 tablet_bootstrap.cc:492] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7: No bootstrap required, opened a new log
I20240502 21:25:29.562358 29646 ts_tablet_manager.cc:1367] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7: Time spent bootstrapping tablet: real 0.006s	user 0.000s	sys 0.004s
I20240502 21:25:29.563814 29646 raft_consensus.cc:348] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:29.564126 29646 raft_consensus.cc:374] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:29.564216 29646 raft_consensus.cc:729] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e36797791842e489aee30de261c4c7, State: Initialized, Role: FOLLOWER
I20240502 21:25:29.564450 29646 consensus_queue.cc:260] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:29.566368 29644 tablet_bootstrap.cc:492] T d25783a912884d2ba72c586d6cd7dfdd P 8786a1419d3f4bc79b88c54ce0675aa6: No bootstrap required, opened a new log
I20240502 21:25:29.566543 29644 ts_tablet_manager.cc:1367] T d25783a912884d2ba72c586d6cd7dfdd P 8786a1419d3f4bc79b88c54ce0675aa6: Time spent bootstrapping tablet: real 0.025s	user 0.006s	sys 0.009s
I20240502 21:25:29.567481 29645 tablet_bootstrap.cc:492] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967: No bootstrap required, opened a new log
I20240502 21:25:29.567605 29643 ts_tablet_manager.cc:1398] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0: Time spent starting tablet: real 0.040s	user 0.032s	sys 0.001s
I20240502 21:25:29.567762 29645 ts_tablet_manager.cc:1367] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967: Time spent bootstrapping tablet: real 0.018s	user 0.015s	sys 0.000s
I20240502 21:25:29.567734 29644 raft_consensus.cc:348] T d25783a912884d2ba72c586d6cd7dfdd P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:29.567986 29644 raft_consensus.cc:374] T d25783a912884d2ba72c586d6cd7dfdd P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:29.568094 29643 tablet_bootstrap.cc:492] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0: Bootstrap starting.
I20240502 21:25:29.568094 29644 raft_consensus.cc:729] T d25783a912884d2ba72c586d6cd7dfdd P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8786a1419d3f4bc79b88c54ce0675aa6, State: Initialized, Role: FOLLOWER
I20240502 21:25:29.568367 29644 consensus_queue.cc:260] T d25783a912884d2ba72c586d6cd7dfdd P 8786a1419d3f4bc79b88c54ce0675aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:29.569958 29643 tablet_bootstrap.cc:654] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:29.570144 29653 raft_consensus.cc:482] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:29.570463 29653 raft_consensus.cc:504] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:29.575700 29653 leader_election.cc:288] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795), 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815)
I20240502 21:25:29.576567 29646 ts_tablet_manager.cc:1398] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7: Time spent starting tablet: real 0.014s	user 0.003s	sys 0.000s
I20240502 21:25:29.576622 29645 raft_consensus.cc:348] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:25:29.577527 29645 raft_consensus.cc:374] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:29.577863 29645 raft_consensus.cc:729] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8733381e64da4e91a8a54e23b01e8967, State: Initialized, Role: FOLLOWER
I20240502 21:25:29.577955 29646 tablet_bootstrap.cc:492] T 20322b184e2f4a88b1abb569e6916f7c P 04e36797791842e489aee30de261c4c7: Bootstrap starting.
I20240502 21:25:29.578387 29645 consensus_queue.cc:260] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:25:29.580461 29645 ts_tablet_manager.cc:1398] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967: Time spent starting tablet: real 0.013s	user 0.002s	sys 0.000s
I20240502 21:25:29.581588 29644 ts_tablet_manager.cc:1398] T d25783a912884d2ba72c586d6cd7dfdd P 8786a1419d3f4bc79b88c54ce0675aa6: Time spent starting tablet: real 0.015s	user 0.003s	sys 0.000s
I20240502 21:25:29.581925 29646 tablet_bootstrap.cc:654] T 20322b184e2f4a88b1abb569e6916f7c P 04e36797791842e489aee30de261c4c7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:29.591800 29646 tablet_bootstrap.cc:492] T 20322b184e2f4a88b1abb569e6916f7c P 04e36797791842e489aee30de261c4c7: No bootstrap required, opened a new log
I20240502 21:25:29.592084 29646 ts_tablet_manager.cc:1367] T 20322b184e2f4a88b1abb569e6916f7c P 04e36797791842e489aee30de261c4c7: Time spent bootstrapping tablet: real 0.014s	user 0.007s	sys 0.003s
I20240502 21:25:29.593152 29646 raft_consensus.cc:348] T 20322b184e2f4a88b1abb569e6916f7c P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:25:29.593477 29646 raft_consensus.cc:374] T 20322b184e2f4a88b1abb569e6916f7c P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:29.593556 29646 raft_consensus.cc:729] T 20322b184e2f4a88b1abb569e6916f7c P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e36797791842e489aee30de261c4c7, State: Initialized, Role: FOLLOWER
I20240502 21:25:29.594167 29646 consensus_queue.cc:260] T 20322b184e2f4a88b1abb569e6916f7c P 04e36797791842e489aee30de261c4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:25:29.595049 29646 ts_tablet_manager.cc:1398] T 20322b184e2f4a88b1abb569e6916f7c P 04e36797791842e489aee30de261c4c7: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 21:25:29.598649 29643 tablet_bootstrap.cc:492] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0: No bootstrap required, opened a new log
I20240502 21:25:29.598841 29643 ts_tablet_manager.cc:1367] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0: Time spent bootstrapping tablet: real 0.031s	user 0.013s	sys 0.015s
I20240502 21:25:29.599633 29030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25783a912884d2ba72c586d6cd7dfdd" candidate_uuid: "04e36797791842e489aee30de261c4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c5becf6262415cab0408b3968ed4c0" is_pre_election: true
I20240502 21:25:29.600188 29643 raft_consensus.cc:348] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:29.600440 29643 raft_consensus.cc:374] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:29.600553 29643 raft_consensus.cc:729] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c5becf6262415cab0408b3968ed4c0, State: Initialized, Role: FOLLOWER
I20240502 21:25:29.600669 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25783a912884d2ba72c586d6cd7dfdd" candidate_uuid: "04e36797791842e489aee30de261c4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" is_pre_election: true
I20240502 21:25:29.600840 29643 consensus_queue.cc:260] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:29.601253 28756 raft_consensus.cc:2454] T d25783a912884d2ba72c586d6cd7dfdd P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04e36797791842e489aee30de261c4c7 in term 0.
I20240502 21:25:29.601331 29030 raft_consensus.cc:2454] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04e36797791842e489aee30de261c4c7 in term 0.
I20240502 21:25:29.601478 29643 ts_tablet_manager.cc:1398] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240502 21:25:29.601974 28552 leader_election.cc:302] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04e36797791842e489aee30de261c4c7, 8786a1419d3f4bc79b88c54ce0675aa6; no voters: 
W20240502 21:25:29.602486 28941 tablet.cc:2369] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:29.602516 29658 raft_consensus.cc:2781] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:29.602680 29658 raft_consensus.cc:482] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:29.602819 29658 raft_consensus.cc:3037] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:29.605348 29658 raft_consensus.cc:504] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:29.606482 29658 leader_election.cc:288] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7 [CANDIDATE]: Term 1 election: Requested vote from peers 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795), 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815)
I20240502 21:25:29.606622 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25783a912884d2ba72c586d6cd7dfdd" candidate_uuid: "04e36797791842e489aee30de261c4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8786a1419d3f4bc79b88c54ce0675aa6"
I20240502 21:25:29.606835 28756 raft_consensus.cc:3037] T d25783a912884d2ba72c586d6cd7dfdd P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:29.606855 29030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25783a912884d2ba72c586d6cd7dfdd" candidate_uuid: "04e36797791842e489aee30de261c4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c5becf6262415cab0408b3968ed4c0"
I20240502 21:25:29.607316 29030 raft_consensus.cc:3037] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:29.609148 28756 raft_consensus.cc:2454] T d25783a912884d2ba72c586d6cd7dfdd P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04e36797791842e489aee30de261c4c7 in term 1.
I20240502 21:25:29.609565 29030 raft_consensus.cc:2454] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04e36797791842e489aee30de261c4c7 in term 1.
I20240502 21:25:29.609632 28552 leader_election.cc:302] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04e36797791842e489aee30de261c4c7, 8786a1419d3f4bc79b88c54ce0675aa6; no voters: 
I20240502 21:25:29.610241 29658 raft_consensus.cc:2781] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:29.610455 29658 raft_consensus.cc:686] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7 [term 1 LEADER]: Becoming Leader. State: Replica: 04e36797791842e489aee30de261c4c7, State: Running, Role: LEADER
I20240502 21:25:29.611001 29658 consensus_queue.cc:237] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:29.617156 28475 catalog_manager.cc:5484] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04e36797791842e489aee30de261c4c7 (127.27.196.65). New cstate: current_term: 1 leader_uuid: "04e36797791842e489aee30de261c4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:29.638236 29655 raft_consensus.cc:482] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:29.638605 29655 raft_consensus.cc:504] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:25:29.640741 29655 leader_election.cc:288] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545), 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983)
I20240502 21:25:29.648792 28893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94280a60b6d4902b866ae3c30e40470" candidate_uuid: "52c5becf6262415cab0408b3968ed4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8733381e64da4e91a8a54e23b01e8967" is_pre_election: true
I20240502 21:25:29.649048 28620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94280a60b6d4902b866ae3c30e40470" candidate_uuid: "52c5becf6262415cab0408b3968ed4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04e36797791842e489aee30de261c4c7" is_pre_election: true
I20240502 21:25:29.649484 28893 raft_consensus.cc:2454] T e94280a60b6d4902b866ae3c30e40470 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52c5becf6262415cab0408b3968ed4c0 in term 0.
I20240502 21:25:29.649518 28620 raft_consensus.cc:2454] T e94280a60b6d4902b866ae3c30e40470 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52c5becf6262415cab0408b3968ed4c0 in term 0.
W20240502 21:25:29.649616 28668 tablet.cc:2369] T 20322b184e2f4a88b1abb569e6916f7c P 04e36797791842e489aee30de261c4c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:29.650218 28960 leader_election.cc:302] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 04e36797791842e489aee30de261c4c7, 52c5becf6262415cab0408b3968ed4c0, 8733381e64da4e91a8a54e23b01e8967; no voters: 
I20240502 21:25:29.650702 29655 raft_consensus.cc:2781] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:29.650851 29655 raft_consensus.cc:482] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:29.650943 29655 raft_consensus.cc:3037] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:29.653066 29655 raft_consensus.cc:504] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:25:29.653776 29655 leader_election.cc:288] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545), 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983)
I20240502 21:25:29.654371 28620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94280a60b6d4902b866ae3c30e40470" candidate_uuid: "52c5becf6262415cab0408b3968ed4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04e36797791842e489aee30de261c4c7"
I20240502 21:25:29.654556 28620 raft_consensus.cc:3037] T e94280a60b6d4902b866ae3c30e40470 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:29.655395 28893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94280a60b6d4902b866ae3c30e40470" candidate_uuid: "52c5becf6262415cab0408b3968ed4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8733381e64da4e91a8a54e23b01e8967"
I20240502 21:25:29.655637 28893 raft_consensus.cc:3037] T e94280a60b6d4902b866ae3c30e40470 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:29.657650 28620 raft_consensus.cc:2454] T e94280a60b6d4902b866ae3c30e40470 P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52c5becf6262415cab0408b3968ed4c0 in term 1.
I20240502 21:25:29.658344 28961 leader_election.cc:302] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04e36797791842e489aee30de261c4c7, 52c5becf6262415cab0408b3968ed4c0; no voters: 
I20240502 21:25:29.658737 28893 raft_consensus.cc:2454] T e94280a60b6d4902b866ae3c30e40470 P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52c5becf6262415cab0408b3968ed4c0 in term 1.
I20240502 21:25:29.658918 29655 raft_consensus.cc:2781] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:29.659695 29655 raft_consensus.cc:686] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 52c5becf6262415cab0408b3968ed4c0, State: Running, Role: LEADER
I20240502 21:25:29.660737 29655 consensus_queue.cc:237] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:25:29.667259 28475 catalog_manager.cc:5484] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68). New cstate: current_term: 1 leader_uuid: "52c5becf6262415cab0408b3968ed4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } health_report { overall_health: UNKNOWN } } }
W20240502 21:25:29.689604 29076 tablet.cc:2369] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:29.699723 29652 raft_consensus.cc:482] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:29.699977 29652 raft_consensus.cc:504] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:25:29.701766 29652 leader_election.cc:288] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815), 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983)
I20240502 21:25:29.711977 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20322b184e2f4a88b1abb569e6916f7c" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" is_pre_election: true
I20240502 21:25:29.712342 28756 raft_consensus.cc:2454] T 20322b184e2f4a88b1abb569e6916f7c P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8733381e64da4e91a8a54e23b01e8967 in term 0.
I20240502 21:25:29.712446 28620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20322b184e2f4a88b1abb569e6916f7c" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04e36797791842e489aee30de261c4c7" is_pre_election: true
I20240502 21:25:29.712769 28620 raft_consensus.cc:2454] T 20322b184e2f4a88b1abb569e6916f7c P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8733381e64da4e91a8a54e23b01e8967 in term 0.
I20240502 21:25:29.713021 28824 leader_election.cc:302] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8733381e64da4e91a8a54e23b01e8967, 8786a1419d3f4bc79b88c54ce0675aa6; no voters: 
I20240502 21:25:29.713642 29652 raft_consensus.cc:2781] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:29.713811 29652 raft_consensus.cc:482] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:29.713924 29652 raft_consensus.cc:3037] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:25:29.714522 28804 tablet.cc:2369] T 20322b184e2f4a88b1abb569e6916f7c P 8786a1419d3f4bc79b88c54ce0675aa6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:29.717146 29652 raft_consensus.cc:504] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:25:29.718024 29652 leader_election.cc:288] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 election: Requested vote from peers 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815), 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983)
I20240502 21:25:29.718770 28620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20322b184e2f4a88b1abb569e6916f7c" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04e36797791842e489aee30de261c4c7"
I20240502 21:25:29.718780 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20322b184e2f4a88b1abb569e6916f7c" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8786a1419d3f4bc79b88c54ce0675aa6"
I20240502 21:25:29.719020 28756 raft_consensus.cc:3037] T 20322b184e2f4a88b1abb569e6916f7c P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:29.719038 28620 raft_consensus.cc:3037] T 20322b184e2f4a88b1abb569e6916f7c P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:29.721936 28756 raft_consensus.cc:2454] T 20322b184e2f4a88b1abb569e6916f7c P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8733381e64da4e91a8a54e23b01e8967 in term 1.
I20240502 21:25:29.722397 28620 raft_consensus.cc:2454] T 20322b184e2f4a88b1abb569e6916f7c P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8733381e64da4e91a8a54e23b01e8967 in term 1.
I20240502 21:25:29.722563 28824 leader_election.cc:302] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8733381e64da4e91a8a54e23b01e8967, 8786a1419d3f4bc79b88c54ce0675aa6; no voters: 
I20240502 21:25:29.722980 29652 raft_consensus.cc:2781] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:29.723664 29652 raft_consensus.cc:686] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Becoming Leader. State: Replica: 8733381e64da4e91a8a54e23b01e8967, State: Running, Role: LEADER
I20240502 21:25:29.724277 29652 consensus_queue.cc:237] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:25:29.729946 28475 catalog_manager.cc:5484] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67). New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:30.015523 29658 consensus_queue.cc:1035] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:30.028255 29658 consensus_queue.cc:1035] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:30.144228 29655 consensus_queue.cc:1035] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:30.154126 29655 consensus_queue.cc:1035] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:30.179457 29672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:30.179842 29672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:30.188863 29672 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 21:25:30.280050 29652 consensus_queue.cc:1035] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:30.297403 29707 consensus_queue.cc:1035] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s

INSERT report
    rows total: 2000
    time total: 271.818 ms
  time per row: 0.135909 ms
I20240502 21:25:30.625231 28474 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49308:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 21:25:30.637715 28736 tablet_service.cc:1467] Processing CreateTablet for tablet c2cdd0cd6e344ca391d295f92c6f6658 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a4cb53742e604e37ae1a6151409c162b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:30.638015 29008 tablet_service.cc:1467] Processing CreateTablet for tablet c2cdd0cd6e344ca391d295f92c6f6658 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a4cb53742e604e37ae1a6151409c162b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:30.638274 28736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2cdd0cd6e344ca391d295f92c6f6658. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.638463 29008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2cdd0cd6e344ca391d295f92c6f6658. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.638857 28871 tablet_service.cc:1467] Processing CreateTablet for tablet c2cdd0cd6e344ca391d295f92c6f6658 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a4cb53742e604e37ae1a6151409c162b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:30.639575 28871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2cdd0cd6e344ca391d295f92c6f6658. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.639940 29007 tablet_service.cc:1467] Processing CreateTablet for tablet 5370969bf2f44ee0833fafa4285edf9f (DEFAULT_TABLE table=rebalance_test_table_1 [id=a4cb53742e604e37ae1a6151409c162b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:30.640328 29007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5370969bf2f44ee0833fafa4285edf9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.641928 28597 tablet_service.cc:1467] Processing CreateTablet for tablet 82338a2c110b44efa16b202c0bad760c (DEFAULT_TABLE table=rebalance_test_table_1 [id=a4cb53742e604e37ae1a6151409c162b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:30.642371 28597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82338a2c110b44efa16b202c0bad760c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.643069 28873 tablet_service.cc:1467] Processing CreateTablet for tablet 5370969bf2f44ee0833fafa4285edf9f (DEFAULT_TABLE table=rebalance_test_table_1 [id=a4cb53742e604e37ae1a6151409c162b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:30.643404 28870 tablet_service.cc:1467] Processing CreateTablet for tablet 82338a2c110b44efa16b202c0bad760c (DEFAULT_TABLE table=rebalance_test_table_1 [id=a4cb53742e604e37ae1a6151409c162b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:30.643615 28873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5370969bf2f44ee0833fafa4285edf9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.644026 28870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82338a2c110b44efa16b202c0bad760c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.644337 28734 tablet_service.cc:1467] Processing CreateTablet for tablet 82338a2c110b44efa16b202c0bad760c (DEFAULT_TABLE table=rebalance_test_table_1 [id=a4cb53742e604e37ae1a6151409c162b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:30.644873 28734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82338a2c110b44efa16b202c0bad760c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.646257 28735 tablet_service.cc:1467] Processing CreateTablet for tablet 5370969bf2f44ee0833fafa4285edf9f (DEFAULT_TABLE table=rebalance_test_table_1 [id=a4cb53742e604e37ae1a6151409c162b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:30.646817 28735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5370969bf2f44ee0833fafa4285edf9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.658610 29743 tablet_bootstrap.cc:492] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0: Bootstrap starting.
I20240502 21:25:30.660795 29742 tablet_bootstrap.cc:492] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6: Bootstrap starting.
I20240502 21:25:30.663342 29742 tablet_bootstrap.cc:654] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.664086 29743 tablet_bootstrap.cc:654] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.664381 29740 tablet_bootstrap.cc:492] T 82338a2c110b44efa16b202c0bad760c P 04e36797791842e489aee30de261c4c7: Bootstrap starting.
I20240502 21:25:30.665625 29742 tablet_bootstrap.cc:492] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6: No bootstrap required, opened a new log
I20240502 21:25:30.665853 29742 ts_tablet_manager.cc:1367] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.004s
I20240502 21:25:30.666870 29740 tablet_bootstrap.cc:654] T 82338a2c110b44efa16b202c0bad760c P 04e36797791842e489aee30de261c4c7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.666769 29742 raft_consensus.cc:348] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:30.666985 29742 raft_consensus.cc:374] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.667101 29743 tablet_bootstrap.cc:492] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0: No bootstrap required, opened a new log
I20240502 21:25:30.667150 29742 raft_consensus.cc:729] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8786a1419d3f4bc79b88c54ce0675aa6, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.667296 29743 ts_tablet_manager.cc:1367] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0: Time spent bootstrapping tablet: real 0.009s	user 0.001s	sys 0.004s
I20240502 21:25:30.667413 29742 consensus_queue.cc:260] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:30.668082 29742 ts_tablet_manager.cc:1398] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240502 21:25:30.668187 29743 raft_consensus.cc:348] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:30.668403 29743 raft_consensus.cc:374] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.668439 29741 tablet_bootstrap.cc:492] T 5370969bf2f44ee0833fafa4285edf9f P 8733381e64da4e91a8a54e23b01e8967: Bootstrap starting.
I20240502 21:25:30.668519 29743 raft_consensus.cc:729] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c5becf6262415cab0408b3968ed4c0, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.668810 29743 consensus_queue.cc:260] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:30.669085 29742 tablet_bootstrap.cc:492] T 5370969bf2f44ee0833fafa4285edf9f P 8786a1419d3f4bc79b88c54ce0675aa6: Bootstrap starting.
I20240502 21:25:30.670373 29741 tablet_bootstrap.cc:654] T 5370969bf2f44ee0833fafa4285edf9f P 8733381e64da4e91a8a54e23b01e8967: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.670683 29742 tablet_bootstrap.cc:654] T 5370969bf2f44ee0833fafa4285edf9f P 8786a1419d3f4bc79b88c54ce0675aa6: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.671082 29743 ts_tablet_manager.cc:1398] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 21:25:30.672181 29740 tablet_bootstrap.cc:492] T 82338a2c110b44efa16b202c0bad760c P 04e36797791842e489aee30de261c4c7: No bootstrap required, opened a new log
I20240502 21:25:30.672382 29740 ts_tablet_manager.cc:1367] T 82338a2c110b44efa16b202c0bad760c P 04e36797791842e489aee30de261c4c7: Time spent bootstrapping tablet: real 0.008s	user 0.004s	sys 0.000s
I20240502 21:25:30.673348 29740 raft_consensus.cc:348] T 82338a2c110b44efa16b202c0bad760c P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:30.673681 29740 raft_consensus.cc:374] T 82338a2c110b44efa16b202c0bad760c P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.673839 29740 raft_consensus.cc:729] T 82338a2c110b44efa16b202c0bad760c P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e36797791842e489aee30de261c4c7, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.674227 29740 consensus_queue.cc:260] T 82338a2c110b44efa16b202c0bad760c P 04e36797791842e489aee30de261c4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:30.674392 29743 tablet_bootstrap.cc:492] T c2cdd0cd6e344ca391d295f92c6f6658 P 52c5becf6262415cab0408b3968ed4c0: Bootstrap starting.
I20240502 21:25:30.675042 29740 ts_tablet_manager.cc:1398] T 82338a2c110b44efa16b202c0bad760c P 04e36797791842e489aee30de261c4c7: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:25:30.676980 29743 tablet_bootstrap.cc:654] T c2cdd0cd6e344ca391d295f92c6f6658 P 52c5becf6262415cab0408b3968ed4c0: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.677428 29651 raft_consensus.cc:482] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:30.677661 29651 raft_consensus.cc:504] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:30.678934 29742 tablet_bootstrap.cc:492] T 5370969bf2f44ee0833fafa4285edf9f P 8786a1419d3f4bc79b88c54ce0675aa6: No bootstrap required, opened a new log
I20240502 21:25:30.679186 29742 ts_tablet_manager.cc:1367] T 5370969bf2f44ee0833fafa4285edf9f P 8786a1419d3f4bc79b88c54ce0675aa6: Time spent bootstrapping tablet: real 0.010s	user 0.003s	sys 0.005s
I20240502 21:25:30.680215 29742 raft_consensus.cc:348] T 5370969bf2f44ee0833fafa4285edf9f P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:30.680537 29742 raft_consensus.cc:374] T 5370969bf2f44ee0833fafa4285edf9f P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.680676 29742 raft_consensus.cc:729] T 5370969bf2f44ee0833fafa4285edf9f P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8786a1419d3f4bc79b88c54ce0675aa6, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.681013 29742 consensus_queue.cc:260] T 5370969bf2f44ee0833fafa4285edf9f P 8786a1419d3f4bc79b88c54ce0675aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:30.687594 29743 tablet_bootstrap.cc:492] T c2cdd0cd6e344ca391d295f92c6f6658 P 52c5becf6262415cab0408b3968ed4c0: No bootstrap required, opened a new log
I20240502 21:25:30.687969 29743 ts_tablet_manager.cc:1367] T c2cdd0cd6e344ca391d295f92c6f6658 P 52c5becf6262415cab0408b3968ed4c0: Time spent bootstrapping tablet: real 0.017s	user 0.009s	sys 0.002s
I20240502 21:25:30.690229 29743 raft_consensus.cc:348] T c2cdd0cd6e344ca391d295f92c6f6658 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:30.690691 29743 raft_consensus.cc:374] T c2cdd0cd6e344ca391d295f92c6f6658 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.691540 29743 raft_consensus.cc:729] T c2cdd0cd6e344ca391d295f92c6f6658 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c5becf6262415cab0408b3968ed4c0, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.692097 29743 consensus_queue.cc:260] T c2cdd0cd6e344ca391d295f92c6f6658 P 52c5becf6262415cab0408b3968ed4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:30.693420 29651 leader_election.cc:288] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983), 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545)
I20240502 21:25:30.694164 29743 ts_tablet_manager.cc:1398] T c2cdd0cd6e344ca391d295f92c6f6658 P 52c5becf6262415cab0408b3968ed4c0: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240502 21:25:30.705116 29742 ts_tablet_manager.cc:1398] T 5370969bf2f44ee0833fafa4285edf9f P 8786a1419d3f4bc79b88c54ce0675aa6: Time spent starting tablet: real 0.026s	user 0.002s	sys 0.018s
I20240502 21:25:30.705300 29741 tablet_bootstrap.cc:492] T 5370969bf2f44ee0833fafa4285edf9f P 8733381e64da4e91a8a54e23b01e8967: No bootstrap required, opened a new log
I20240502 21:25:30.706281 29742 tablet_bootstrap.cc:492] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6: Bootstrap starting.
I20240502 21:25:30.706425 29741 ts_tablet_manager.cc:1367] T 5370969bf2f44ee0833fafa4285edf9f P 8733381e64da4e91a8a54e23b01e8967: Time spent bootstrapping tablet: real 0.038s	user 0.008s	sys 0.022s
I20240502 21:25:30.707850 29741 raft_consensus.cc:348] T 5370969bf2f44ee0833fafa4285edf9f P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:30.708146 29741 raft_consensus.cc:374] T 5370969bf2f44ee0833fafa4285edf9f P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.708289 29741 raft_consensus.cc:729] T 5370969bf2f44ee0833fafa4285edf9f P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8733381e64da4e91a8a54e23b01e8967, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.708606 29741 consensus_queue.cc:260] T 5370969bf2f44ee0833fafa4285edf9f P 8733381e64da4e91a8a54e23b01e8967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:30.708741 29742 tablet_bootstrap.cc:654] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.709450 29741 ts_tablet_manager.cc:1398] T 5370969bf2f44ee0833fafa4285edf9f P 8733381e64da4e91a8a54e23b01e8967: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 21:25:30.709821 29741 tablet_bootstrap.cc:492] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967: Bootstrap starting.
I20240502 21:25:30.711833 29741 tablet_bootstrap.cc:654] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.715144 28620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82338a2c110b44efa16b202c0bad760c" candidate_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04e36797791842e489aee30de261c4c7" is_pre_election: true
I20240502 21:25:30.715492 28620 raft_consensus.cc:2454] T 82338a2c110b44efa16b202c0bad760c P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8786a1419d3f4bc79b88c54ce0675aa6 in term 0.
I20240502 21:25:30.716063 28893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82338a2c110b44efa16b202c0bad760c" candidate_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8733381e64da4e91a8a54e23b01e8967" is_pre_election: true
I20240502 21:25:30.716645 28687 leader_election.cc:302] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04e36797791842e489aee30de261c4c7, 8786a1419d3f4bc79b88c54ce0675aa6; no voters: 
I20240502 21:25:30.717325 29750 raft_consensus.cc:2781] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:30.717473 29741 tablet_bootstrap.cc:492] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967: No bootstrap required, opened a new log
I20240502 21:25:30.717440 29750 raft_consensus.cc:482] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:30.717520 29742 tablet_bootstrap.cc:492] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6: No bootstrap required, opened a new log
I20240502 21:25:30.717639 29750 raft_consensus.cc:3037] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.717685 29742 ts_tablet_manager.cc:1367] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.002s
I20240502 21:25:30.717695 29741 ts_tablet_manager.cc:1367] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967: Time spent bootstrapping tablet: real 0.008s	user 0.005s	sys 0.000s
I20240502 21:25:30.718694 29742 raft_consensus.cc:348] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
W20240502 21:25:30.718930 28686 leader_election.cc:341] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:30.719199 29741 raft_consensus.cc:348] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:30.719445 29741 raft_consensus.cc:374] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.719530 29741 raft_consensus.cc:729] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8733381e64da4e91a8a54e23b01e8967, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.718959 29742 raft_consensus.cc:374] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.719848 29741 consensus_queue.cc:260] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:30.720041 29742 raft_consensus.cc:729] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8786a1419d3f4bc79b88c54ce0675aa6, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.720330 29742 consensus_queue.cc:260] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:30.720561 29741 ts_tablet_manager.cc:1398] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 21:25:30.721041 29742 ts_tablet_manager.cc:1398] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.002s
I20240502 21:25:30.721652 29741 tablet_bootstrap.cc:492] T 82338a2c110b44efa16b202c0bad760c P 8733381e64da4e91a8a54e23b01e8967: Bootstrap starting.
I20240502 21:25:30.723861 29741 tablet_bootstrap.cc:654] T 82338a2c110b44efa16b202c0bad760c P 8733381e64da4e91a8a54e23b01e8967: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.721283 29750 raft_consensus.cc:504] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:30.727486 29750 leader_election.cc:288] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 1 election: Requested vote from peers 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983), 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545)
I20240502 21:25:30.728552 28893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82338a2c110b44efa16b202c0bad760c" candidate_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8733381e64da4e91a8a54e23b01e8967"
I20240502 21:25:30.729032 28620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82338a2c110b44efa16b202c0bad760c" candidate_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04e36797791842e489aee30de261c4c7"
I20240502 21:25:30.729355 28620 raft_consensus.cc:3037] T 82338a2c110b44efa16b202c0bad760c P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:25:30.729384 28686 leader_election.cc:341] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:30.729811 29741 tablet_bootstrap.cc:492] T 82338a2c110b44efa16b202c0bad760c P 8733381e64da4e91a8a54e23b01e8967: No bootstrap required, opened a new log
I20240502 21:25:30.729943 29741 ts_tablet_manager.cc:1367] T 82338a2c110b44efa16b202c0bad760c P 8733381e64da4e91a8a54e23b01e8967: Time spent bootstrapping tablet: real 0.008s	user 0.006s	sys 0.000s
I20240502 21:25:30.730801 29741 raft_consensus.cc:348] T 82338a2c110b44efa16b202c0bad760c P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:30.730965 29741 raft_consensus.cc:374] T 82338a2c110b44efa16b202c0bad760c P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.731029 29741 raft_consensus.cc:729] T 82338a2c110b44efa16b202c0bad760c P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8733381e64da4e91a8a54e23b01e8967, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.731309 29741 consensus_queue.cc:260] T 82338a2c110b44efa16b202c0bad760c P 8733381e64da4e91a8a54e23b01e8967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:30.731961 29741 ts_tablet_manager.cc:1398] T 82338a2c110b44efa16b202c0bad760c P 8733381e64da4e91a8a54e23b01e8967: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:25:30.732892 28620 raft_consensus.cc:2454] T 82338a2c110b44efa16b202c0bad760c P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8786a1419d3f4bc79b88c54ce0675aa6 in term 1.
I20240502 21:25:30.733775 28687 leader_election.cc:302] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04e36797791842e489aee30de261c4c7, 8786a1419d3f4bc79b88c54ce0675aa6; no voters: 8733381e64da4e91a8a54e23b01e8967
I20240502 21:25:30.734432 29750 raft_consensus.cc:2781] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:30.734656 29750 raft_consensus.cc:686] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 LEADER]: Becoming Leader. State: Replica: 8786a1419d3f4bc79b88c54ce0675aa6, State: Running, Role: LEADER
I20240502 21:25:30.735350 29750 consensus_queue.cc:237] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:30.739811 29655 raft_consensus.cc:482] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:30.740016 29655 raft_consensus.cc:504] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:30.740886 29655 leader_election.cc:288] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815), 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545)
I20240502 21:25:30.740887 28473 catalog_manager.cc:5484] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:30.741744 28893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5370969bf2f44ee0833fafa4285edf9f" candidate_uuid: "52c5becf6262415cab0408b3968ed4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8733381e64da4e91a8a54e23b01e8967" is_pre_election: true
I20240502 21:25:30.742028 28893 raft_consensus.cc:2454] T 5370969bf2f44ee0833fafa4285edf9f P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52c5becf6262415cab0408b3968ed4c0 in term 0.
I20240502 21:25:30.742583 28960 leader_election.cc:302] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52c5becf6262415cab0408b3968ed4c0, 8733381e64da4e91a8a54e23b01e8967; no voters: 
I20240502 21:25:30.742599 29706 raft_consensus.cc:482] T 82338a2c110b44efa16b202c0bad760c P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:30.742777 29706 raft_consensus.cc:504] T 82338a2c110b44efa16b202c0bad760c P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:30.742988 29655 raft_consensus.cc:2781] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:30.743088 29655 raft_consensus.cc:482] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:30.743208 29655 raft_consensus.cc:3037] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.743815 29706 leader_election.cc:288] T 82338a2c110b44efa16b202c0bad760c P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983), 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815)
I20240502 21:25:30.747866 29655 raft_consensus.cc:504] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:30.748122 28620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82338a2c110b44efa16b202c0bad760c" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04e36797791842e489aee30de261c4c7" is_pre_election: true
I20240502 21:25:30.748622 28620 raft_consensus.cc:2379] T 82338a2c110b44efa16b202c0bad760c P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8733381e64da4e91a8a54e23b01e8967 in current term 1: Already voted for candidate 8786a1419d3f4bc79b88c54ce0675aa6 in this term.
I20240502 21:25:30.748732 29655 leader_election.cc:288] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815), 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545)
I20240502 21:25:30.749565 28893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5370969bf2f44ee0833fafa4285edf9f" candidate_uuid: "52c5becf6262415cab0408b3968ed4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8733381e64da4e91a8a54e23b01e8967"
I20240502 21:25:30.749847 28893 raft_consensus.cc:3037] T 5370969bf2f44ee0833fafa4285edf9f P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.750340 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82338a2c110b44efa16b202c0bad760c" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" is_pre_election: true
I20240502 21:25:30.751060 28824 leader_election.cc:302] T 82338a2c110b44efa16b202c0bad760c P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8733381e64da4e91a8a54e23b01e8967; no voters: 04e36797791842e489aee30de261c4c7, 8786a1419d3f4bc79b88c54ce0675aa6
I20240502 21:25:30.751457 29706 raft_consensus.cc:3037] T 82338a2c110b44efa16b202c0bad760c P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.752651 28893 raft_consensus.cc:2454] T 5370969bf2f44ee0833fafa4285edf9f P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52c5becf6262415cab0408b3968ed4c0 in term 1.
I20240502 21:25:30.753243 28960 leader_election.cc:302] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52c5becf6262415cab0408b3968ed4c0, 8733381e64da4e91a8a54e23b01e8967; no voters: 
I20240502 21:25:30.753547 29655 raft_consensus.cc:2781] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:30.753708 29655 raft_consensus.cc:686] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 52c5becf6262415cab0408b3968ed4c0, State: Running, Role: LEADER
I20240502 21:25:30.754029 29655 consensus_queue.cc:237] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:30.755419 29706 raft_consensus.cc:2726] T 82338a2c110b44efa16b202c0bad760c P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:25:30.758450 28473 catalog_manager.cc:5484] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68). New cstate: current_term: 1 leader_uuid: "52c5becf6262415cab0408b3968ed4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:30.759696 28755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5370969bf2f44ee0833fafa4285edf9f" candidate_uuid: "52c5becf6262415cab0408b3968ed4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8786a1419d3f4bc79b88c54ce0675aa6"
I20240502 21:25:30.759734 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5370969bf2f44ee0833fafa4285edf9f" candidate_uuid: "52c5becf6262415cab0408b3968ed4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" is_pre_election: true
I20240502 21:25:30.759894 28755 raft_consensus.cc:3037] T 5370969bf2f44ee0833fafa4285edf9f P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.762446 28755 raft_consensus.cc:2454] T 5370969bf2f44ee0833fafa4285edf9f P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52c5becf6262415cab0408b3968ed4c0 in term 1.
I20240502 21:25:30.824443 29706 raft_consensus.cc:482] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:30.824836 29706 raft_consensus.cc:504] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:30.825927 29706 leader_election.cc:288] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815), 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795)
I20240502 21:25:30.826774 28755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2cdd0cd6e344ca391d295f92c6f6658" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" is_pre_election: true
I20240502 21:25:30.827204 28755 raft_consensus.cc:2454] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8733381e64da4e91a8a54e23b01e8967 in term 0.
I20240502 21:25:30.827950 28824 leader_election.cc:302] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8733381e64da4e91a8a54e23b01e8967, 8786a1419d3f4bc79b88c54ce0675aa6; no voters: 
I20240502 21:25:30.828502 29706 raft_consensus.cc:2781] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:30.828768 29706 raft_consensus.cc:482] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:30.829007 29706 raft_consensus.cc:3037] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.832118 29706 raft_consensus.cc:504] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:30.833225 29706 leader_election.cc:288] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 election: Requested vote from peers 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815), 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795)
I20240502 21:25:30.833936 28755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2cdd0cd6e344ca391d295f92c6f6658" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8786a1419d3f4bc79b88c54ce0675aa6"
I20240502 21:25:30.834268 28755 raft_consensus.cc:3037] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.835779 29029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2cdd0cd6e344ca391d295f92c6f6658" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c5becf6262415cab0408b3968ed4c0" is_pre_election: true
I20240502 21:25:30.835920 29030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2cdd0cd6e344ca391d295f92c6f6658" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c5becf6262415cab0408b3968ed4c0"
I20240502 21:25:30.836097 29029 raft_consensus.cc:2454] T c2cdd0cd6e344ca391d295f92c6f6658 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8733381e64da4e91a8a54e23b01e8967 in term 0.
I20240502 21:25:30.836985 28755 raft_consensus.cc:2454] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8733381e64da4e91a8a54e23b01e8967 in term 1.
I20240502 21:25:30.837401 28824 leader_election.cc:302] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8733381e64da4e91a8a54e23b01e8967, 8786a1419d3f4bc79b88c54ce0675aa6; no voters: 52c5becf6262415cab0408b3968ed4c0
I20240502 21:25:30.837841 29706 raft_consensus.cc:2781] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:30.838099 29706 raft_consensus.cc:686] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Becoming Leader. State: Replica: 8733381e64da4e91a8a54e23b01e8967, State: Running, Role: LEADER
I20240502 21:25:30.838498 29706 consensus_queue.cc:237] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:30.842558 28473 catalog_manager.cc:5484] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67). New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } health_report { overall_health: HEALTHY } } }
W20240502 21:25:31.244669 29762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:31.245085 29762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:31.253300 29762 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:25:31.265292 29766 consensus_queue.cc:1035] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:31.268776 29766 consensus_queue.cc:1035] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:31.310493 29652 consensus_queue.cc:1035] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:31.315990 29029 raft_consensus.cc:3037] T c2cdd0cd6e344ca391d295f92c6f6658 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Advancing to term 1
Using table 'rebalance_test_table_1'
I20240502 21:25:31.320529 29719 consensus_queue.cc:1035] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:31.359261 29655 consensus_queue.cc:1035] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:31.396272 29797 consensus_queue.cc:1035] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 336.903 ms
  time per row: 0.168452 ms
I20240502 21:25:31.788204 28474 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49308:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 21:25:31.799849 28597 tablet_service.cc:1467] Processing CreateTablet for tablet 876004fec43e428e930f3fbb873ccac8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=125451552e264fde833414d8cda8be18]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:31.799921 29007 tablet_service.cc:1467] Processing CreateTablet for tablet 876004fec43e428e930f3fbb873ccac8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=125451552e264fde833414d8cda8be18]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:31.800359 28735 tablet_service.cc:1467] Processing CreateTablet for tablet 876004fec43e428e930f3fbb873ccac8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=125451552e264fde833414d8cda8be18]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:31.800508 29007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 876004fec43e428e930f3fbb873ccac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:31.800540 28597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 876004fec43e428e930f3fbb873ccac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:31.800835 28735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 876004fec43e428e930f3fbb873ccac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:31.800913 28873 tablet_service.cc:1467] Processing CreateTablet for tablet 4a58681ae3e54019aefa430aaa3dc046 (DEFAULT_TABLE table=rebalance_test_table_2 [id=125451552e264fde833414d8cda8be18]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:31.801393 28873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a58681ae3e54019aefa430aaa3dc046. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:31.806694 29008 tablet_service.cc:1467] Processing CreateTablet for tablet 4a58681ae3e54019aefa430aaa3dc046 (DEFAULT_TABLE table=rebalance_test_table_2 [id=125451552e264fde833414d8cda8be18]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:31.807202 29008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a58681ae3e54019aefa430aaa3dc046. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:31.808365 28599 tablet_service.cc:1467] Processing CreateTablet for tablet 4a58681ae3e54019aefa430aaa3dc046 (DEFAULT_TABLE table=rebalance_test_table_2 [id=125451552e264fde833414d8cda8be18]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:31.809150 28599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a58681ae3e54019aefa430aaa3dc046. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:31.810962 28736 tablet_service.cc:1467] Processing CreateTablet for tablet a604aabc523640089bc54cd6473a9a95 (DEFAULT_TABLE table=rebalance_test_table_2 [id=125451552e264fde833414d8cda8be18]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:31.811483 28736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a604aabc523640089bc54cd6473a9a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:31.813261 29006 tablet_service.cc:1467] Processing CreateTablet for tablet a604aabc523640089bc54cd6473a9a95 (DEFAULT_TABLE table=rebalance_test_table_2 [id=125451552e264fde833414d8cda8be18]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:31.813737 29006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a604aabc523640089bc54cd6473a9a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:31.816128 28598 tablet_service.cc:1467] Processing CreateTablet for tablet a604aabc523640089bc54cd6473a9a95 (DEFAULT_TABLE table=rebalance_test_table_2 [id=125451552e264fde833414d8cda8be18]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:31.816601 28598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a604aabc523640089bc54cd6473a9a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:31.822543 29837 tablet_bootstrap.cc:492] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967: Bootstrap starting.
I20240502 21:25:31.824753 29837 tablet_bootstrap.cc:654] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:31.826361 29836 tablet_bootstrap.cc:492] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6: Bootstrap starting.
I20240502 21:25:31.826942 29838 tablet_bootstrap.cc:492] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0: Bootstrap starting.
I20240502 21:25:31.827420 29837 tablet_bootstrap.cc:492] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967: No bootstrap required, opened a new log
I20240502 21:25:31.827673 29837 ts_tablet_manager.cc:1367] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967: Time spent bootstrapping tablet: real 0.005s	user 0.005s	sys 0.000s
I20240502 21:25:31.829097 29836 tablet_bootstrap.cc:654] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:31.829159 29838 tablet_bootstrap.cc:654] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:31.829658 29837 raft_consensus.cc:348] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:31.829957 29837 raft_consensus.cc:374] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:31.830062 29837 raft_consensus.cc:729] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8733381e64da4e91a8a54e23b01e8967, State: Initialized, Role: FOLLOWER
I20240502 21:25:31.830458 29837 consensus_queue.cc:260] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:31.830709 29839 tablet_bootstrap.cc:492] T 4a58681ae3e54019aefa430aaa3dc046 P 04e36797791842e489aee30de261c4c7: Bootstrap starting.
I20240502 21:25:31.831449 29837 ts_tablet_manager.cc:1398] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 21:25:31.833066 29838 tablet_bootstrap.cc:492] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0: No bootstrap required, opened a new log
I20240502 21:25:31.833381 29838 ts_tablet_manager.cc:1367] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0: Time spent bootstrapping tablet: real 0.007s	user 0.004s	sys 0.000s
I20240502 21:25:31.833494 29839 tablet_bootstrap.cc:654] T 4a58681ae3e54019aefa430aaa3dc046 P 04e36797791842e489aee30de261c4c7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:31.834652 29838 raft_consensus.cc:348] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:31.835058 29838 raft_consensus.cc:374] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:31.835276 29838 raft_consensus.cc:729] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c5becf6262415cab0408b3968ed4c0, State: Initialized, Role: FOLLOWER
I20240502 21:25:31.835711 29838 consensus_queue.cc:260] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:31.836746 29838 ts_tablet_manager.cc:1398] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 21:25:31.837255 29838 tablet_bootstrap.cc:492] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0: Bootstrap starting.
I20240502 21:25:31.840741 29838 tablet_bootstrap.cc:654] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:31.843504 29839 tablet_bootstrap.cc:492] T 4a58681ae3e54019aefa430aaa3dc046 P 04e36797791842e489aee30de261c4c7: No bootstrap required, opened a new log
I20240502 21:25:31.843714 29839 ts_tablet_manager.cc:1367] T 4a58681ae3e54019aefa430aaa3dc046 P 04e36797791842e489aee30de261c4c7: Time spent bootstrapping tablet: real 0.013s	user 0.003s	sys 0.003s
I20240502 21:25:31.843964 29836 tablet_bootstrap.cc:492] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6: No bootstrap required, opened a new log
I20240502 21:25:31.844146 29836 ts_tablet_manager.cc:1367] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6: Time spent bootstrapping tablet: real 0.018s	user 0.006s	sys 0.006s
I20240502 21:25:31.844421 29838 tablet_bootstrap.cc:492] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0: No bootstrap required, opened a new log
I20240502 21:25:31.844624 29838 ts_tablet_manager.cc:1367] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.000s
I20240502 21:25:31.844635 29839 raft_consensus.cc:348] T 4a58681ae3e54019aefa430aaa3dc046 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:31.844859 29839 raft_consensus.cc:374] T 4a58681ae3e54019aefa430aaa3dc046 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:31.844995 29839 raft_consensus.cc:729] T 4a58681ae3e54019aefa430aaa3dc046 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e36797791842e489aee30de261c4c7, State: Initialized, Role: FOLLOWER
I20240502 21:25:31.845142 29836 raft_consensus.cc:348] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:31.845275 29839 consensus_queue.cc:260] T 4a58681ae3e54019aefa430aaa3dc046 P 04e36797791842e489aee30de261c4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:31.845456 29836 raft_consensus.cc:374] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:31.845583 29836 raft_consensus.cc:729] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8786a1419d3f4bc79b88c54ce0675aa6, State: Initialized, Role: FOLLOWER
I20240502 21:25:31.845669 29838 raft_consensus.cc:348] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:31.845916 29838 raft_consensus.cc:374] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:31.845959 29839 ts_tablet_manager.cc:1398] T 4a58681ae3e54019aefa430aaa3dc046 P 04e36797791842e489aee30de261c4c7: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 21:25:31.845839 29836 consensus_queue.cc:260] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:31.845988 29838 raft_consensus.cc:729] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c5becf6262415cab0408b3968ed4c0, State: Initialized, Role: FOLLOWER
I20240502 21:25:31.846282 29838 consensus_queue.cc:260] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:31.846974 29838 ts_tablet_manager.cc:1398] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 21:25:31.847409 29838 tablet_bootstrap.cc:492] T a604aabc523640089bc54cd6473a9a95 P 52c5becf6262415cab0408b3968ed4c0: Bootstrap starting.
I20240502 21:25:31.847930 29836 ts_tablet_manager.cc:1398] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.001s
I20240502 21:25:31.848286 29836 tablet_bootstrap.cc:492] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6: Bootstrap starting.
I20240502 21:25:31.848413 29839 tablet_bootstrap.cc:492] T 876004fec43e428e930f3fbb873ccac8 P 04e36797791842e489aee30de261c4c7: Bootstrap starting.
I20240502 21:25:31.849311 29838 tablet_bootstrap.cc:654] T a604aabc523640089bc54cd6473a9a95 P 52c5becf6262415cab0408b3968ed4c0: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:31.849747 29800 raft_consensus.cc:482] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:31.849939 29800 raft_consensus.cc:504] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:31.850144 29836 tablet_bootstrap.cc:654] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:31.850821 29800 leader_election.cc:288] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983), 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815)
I20240502 21:25:31.852416 28755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876004fec43e428e930f3fbb873ccac8" candidate_uuid: "52c5becf6262415cab0408b3968ed4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" is_pre_election: true
I20240502 21:25:31.852726 28755 raft_consensus.cc:2454] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52c5becf6262415cab0408b3968ed4c0 in term 0.
I20240502 21:25:31.852885 28620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876004fec43e428e930f3fbb873ccac8" candidate_uuid: "52c5becf6262415cab0408b3968ed4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04e36797791842e489aee30de261c4c7" is_pre_election: true
W20240502 21:25:31.853821 28961 leader_election.cc:341] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:31.854163 28961 leader_election.cc:302] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52c5becf6262415cab0408b3968ed4c0, 8786a1419d3f4bc79b88c54ce0675aa6; no voters: 04e36797791842e489aee30de261c4c7
I20240502 21:25:31.854857 29800 raft_consensus.cc:2781] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:31.854990 29800 raft_consensus.cc:482] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:31.855091 29800 raft_consensus.cc:3037] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:31.858217 29800 raft_consensus.cc:504] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:31.859001 29800 leader_election.cc:288] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983), 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815)
I20240502 21:25:31.860018 28755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876004fec43e428e930f3fbb873ccac8" candidate_uuid: "52c5becf6262415cab0408b3968ed4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8786a1419d3f4bc79b88c54ce0675aa6"
I20240502 21:25:31.860272 28755 raft_consensus.cc:3037] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:31.860389 29836 tablet_bootstrap.cc:492] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6: No bootstrap required, opened a new log
I20240502 21:25:31.860631 29836 ts_tablet_manager.cc:1367] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6: Time spent bootstrapping tablet: real 0.012s	user 0.010s	sys 0.000s
I20240502 21:25:31.861142 28620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876004fec43e428e930f3fbb873ccac8" candidate_uuid: "52c5becf6262415cab0408b3968ed4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04e36797791842e489aee30de261c4c7"
I20240502 21:25:31.861907 29836 raft_consensus.cc:348] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:31.862231 29836 raft_consensus.cc:374] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:31.862325 29836 raft_consensus.cc:729] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8786a1419d3f4bc79b88c54ce0675aa6, State: Initialized, Role: FOLLOWER
I20240502 21:25:31.862715 29836 consensus_queue.cc:260] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:31.863651 29836 ts_tablet_manager.cc:1398] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
W20240502 21:25:31.864372 28961 leader_election.cc:341] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:31.865612 28755 raft_consensus.cc:2454] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52c5becf6262415cab0408b3968ed4c0 in term 1.
I20240502 21:25:31.866283 28961 leader_election.cc:302] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52c5becf6262415cab0408b3968ed4c0, 8786a1419d3f4bc79b88c54ce0675aa6; no voters: 04e36797791842e489aee30de261c4c7
I20240502 21:25:31.866856 29798 raft_consensus.cc:2781] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:31.866990 29798 raft_consensus.cc:686] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 52c5becf6262415cab0408b3968ed4c0, State: Running, Role: LEADER
I20240502 21:25:31.867098 29839 tablet_bootstrap.cc:654] T 876004fec43e428e930f3fbb873ccac8 P 04e36797791842e489aee30de261c4c7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:31.867839 29798 consensus_queue.cc:237] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:31.868803 29838 tablet_bootstrap.cc:492] T a604aabc523640089bc54cd6473a9a95 P 52c5becf6262415cab0408b3968ed4c0: No bootstrap required, opened a new log
I20240502 21:25:31.869015 29838 ts_tablet_manager.cc:1367] T a604aabc523640089bc54cd6473a9a95 P 52c5becf6262415cab0408b3968ed4c0: Time spent bootstrapping tablet: real 0.022s	user 0.004s	sys 0.016s
I20240502 21:25:31.869122 29839 tablet_bootstrap.cc:492] T 876004fec43e428e930f3fbb873ccac8 P 04e36797791842e489aee30de261c4c7: No bootstrap required, opened a new log
I20240502 21:25:31.869360 29839 ts_tablet_manager.cc:1367] T 876004fec43e428e930f3fbb873ccac8 P 04e36797791842e489aee30de261c4c7: Time spent bootstrapping tablet: real 0.021s	user 0.004s	sys 0.001s
I20240502 21:25:31.870059 29838 raft_consensus.cc:348] T a604aabc523640089bc54cd6473a9a95 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:31.870313 29838 raft_consensus.cc:374] T a604aabc523640089bc54cd6473a9a95 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:31.870290 29839 raft_consensus.cc:348] T 876004fec43e428e930f3fbb873ccac8 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:31.870466 29838 raft_consensus.cc:729] T a604aabc523640089bc54cd6473a9a95 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c5becf6262415cab0408b3968ed4c0, State: Initialized, Role: FOLLOWER
I20240502 21:25:31.870535 29839 raft_consensus.cc:374] T 876004fec43e428e930f3fbb873ccac8 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:31.870635 29839 raft_consensus.cc:729] T 876004fec43e428e930f3fbb873ccac8 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e36797791842e489aee30de261c4c7, State: Initialized, Role: FOLLOWER
I20240502 21:25:31.870728 29838 consensus_queue.cc:260] T a604aabc523640089bc54cd6473a9a95 P 52c5becf6262415cab0408b3968ed4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:31.870908 29839 consensus_queue.cc:260] T 876004fec43e428e930f3fbb873ccac8 P 04e36797791842e489aee30de261c4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:31.871376 29838 ts_tablet_manager.cc:1398] T a604aabc523640089bc54cd6473a9a95 P 52c5becf6262415cab0408b3968ed4c0: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 21:25:31.871541 29839 ts_tablet_manager.cc:1398] T 876004fec43e428e930f3fbb873ccac8 P 04e36797791842e489aee30de261c4c7: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:25:31.871475 28474 catalog_manager.cc:5484] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68). New cstate: current_term: 1 leader_uuid: "52c5becf6262415cab0408b3968ed4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:31.871850 29839 tablet_bootstrap.cc:492] T a604aabc523640089bc54cd6473a9a95 P 04e36797791842e489aee30de261c4c7: Bootstrap starting.
I20240502 21:25:31.873816 29839 tablet_bootstrap.cc:654] T a604aabc523640089bc54cd6473a9a95 P 04e36797791842e489aee30de261c4c7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:31.876595 29839 tablet_bootstrap.cc:492] T a604aabc523640089bc54cd6473a9a95 P 04e36797791842e489aee30de261c4c7: No bootstrap required, opened a new log
I20240502 21:25:31.876775 29839 ts_tablet_manager.cc:1367] T a604aabc523640089bc54cd6473a9a95 P 04e36797791842e489aee30de261c4c7: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 21:25:31.877425 29839 raft_consensus.cc:348] T a604aabc523640089bc54cd6473a9a95 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:31.877674 29839 raft_consensus.cc:374] T a604aabc523640089bc54cd6473a9a95 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:31.877830 29839 raft_consensus.cc:729] T a604aabc523640089bc54cd6473a9a95 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e36797791842e489aee30de261c4c7, State: Initialized, Role: FOLLOWER
I20240502 21:25:31.878077 29839 consensus_queue.cc:260] T a604aabc523640089bc54cd6473a9a95 P 04e36797791842e489aee30de261c4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:31.878607 29839 ts_tablet_manager.cc:1398] T a604aabc523640089bc54cd6473a9a95 P 04e36797791842e489aee30de261c4c7: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:25:31.973459 29776 raft_consensus.cc:482] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:31.973686 29776 raft_consensus.cc:504] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:31.974516 29776 leader_election.cc:288] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983), 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795)
I20240502 21:25:31.975263 28620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a604aabc523640089bc54cd6473a9a95" candidate_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04e36797791842e489aee30de261c4c7" is_pre_election: true
I20240502 21:25:31.975586 28620 raft_consensus.cc:2454] T a604aabc523640089bc54cd6473a9a95 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8786a1419d3f4bc79b88c54ce0675aa6 in term 0.
I20240502 21:25:31.976071 28687 leader_election.cc:302] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04e36797791842e489aee30de261c4c7, 8786a1419d3f4bc79b88c54ce0675aa6; no voters: 
I20240502 21:25:31.976390 29776 raft_consensus.cc:2781] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:31.976527 29776 raft_consensus.cc:482] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:31.976639 29776 raft_consensus.cc:3037] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:31.978777 29776 raft_consensus.cc:504] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:31.979527 29776 leader_election.cc:288] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 1 election: Requested vote from peers 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983), 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795)
I20240502 21:25:31.980084 28620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a604aabc523640089bc54cd6473a9a95" candidate_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04e36797791842e489aee30de261c4c7"
I20240502 21:25:31.980355 28620 raft_consensus.cc:3037] T a604aabc523640089bc54cd6473a9a95 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:31.981645 29029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a604aabc523640089bc54cd6473a9a95" candidate_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c5becf6262415cab0408b3968ed4c0" is_pre_election: true
I20240502 21:25:31.981827 29030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a604aabc523640089bc54cd6473a9a95" candidate_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c5becf6262415cab0408b3968ed4c0"
I20240502 21:25:31.981961 29029 raft_consensus.cc:2454] T a604aabc523640089bc54cd6473a9a95 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8786a1419d3f4bc79b88c54ce0675aa6 in term 0.
I20240502 21:25:31.982812 28620 raft_consensus.cc:2454] T a604aabc523640089bc54cd6473a9a95 P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8786a1419d3f4bc79b88c54ce0675aa6 in term 1.
I20240502 21:25:31.983470 28687 leader_election.cc:302] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04e36797791842e489aee30de261c4c7, 8786a1419d3f4bc79b88c54ce0675aa6; no voters: 52c5becf6262415cab0408b3968ed4c0
I20240502 21:25:31.983865 29776 raft_consensus.cc:2781] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:31.984055 29776 raft_consensus.cc:686] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 LEADER]: Becoming Leader. State: Replica: 8786a1419d3f4bc79b88c54ce0675aa6, State: Running, Role: LEADER
I20240502 21:25:31.984433 29776 consensus_queue.cc:237] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:31.987620 28474 catalog_manager.cc:5484] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:32.041365 29706 raft_consensus.cc:482] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:32.041595 29706 raft_consensus.cc:504] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:32.042282 29706 leader_election.cc:288] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795), 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983)
I20240502 21:25:32.043270 28620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a58681ae3e54019aefa430aaa3dc046" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04e36797791842e489aee30de261c4c7" is_pre_election: true
I20240502 21:25:32.043239 29029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a58681ae3e54019aefa430aaa3dc046" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c5becf6262415cab0408b3968ed4c0" is_pre_election: true
I20240502 21:25:32.043633 29029 raft_consensus.cc:2454] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8733381e64da4e91a8a54e23b01e8967 in term 0.
I20240502 21:25:32.043609 28620 raft_consensus.cc:2454] T 4a58681ae3e54019aefa430aaa3dc046 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8733381e64da4e91a8a54e23b01e8967 in term 0.
I20240502 21:25:32.044274 28824 leader_election.cc:302] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04e36797791842e489aee30de261c4c7, 8733381e64da4e91a8a54e23b01e8967; no voters: 
I20240502 21:25:32.044673 29706 raft_consensus.cc:2781] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:32.044839 29706 raft_consensus.cc:482] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:32.044932 29706 raft_consensus.cc:3037] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:32.047354 29706 raft_consensus.cc:504] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:32.048019 29706 leader_election.cc:288] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 election: Requested vote from peers 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795), 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983)
I20240502 21:25:32.048784 29029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a58681ae3e54019aefa430aaa3dc046" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c5becf6262415cab0408b3968ed4c0"
I20240502 21:25:32.048796 28620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a58681ae3e54019aefa430aaa3dc046" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04e36797791842e489aee30de261c4c7"
I20240502 21:25:32.049047 28620 raft_consensus.cc:3037] T 4a58681ae3e54019aefa430aaa3dc046 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:32.049095 29029 raft_consensus.cc:3037] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:32.052307 28620 raft_consensus.cc:2454] T 4a58681ae3e54019aefa430aaa3dc046 P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8733381e64da4e91a8a54e23b01e8967 in term 1.
I20240502 21:25:32.052286 29029 raft_consensus.cc:2454] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8733381e64da4e91a8a54e23b01e8967 in term 1.
I20240502 21:25:32.052898 28824 leader_election.cc:302] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04e36797791842e489aee30de261c4c7, 8733381e64da4e91a8a54e23b01e8967; no voters: 
I20240502 21:25:32.053318 29706 raft_consensus.cc:2781] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:32.053467 29706 raft_consensus.cc:686] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Becoming Leader. State: Replica: 8733381e64da4e91a8a54e23b01e8967, State: Running, Role: LEADER
I20240502 21:25:32.053781 29706 consensus_queue.cc:237] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:32.057289 28474 catalog_manager.cc:5484] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67). New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:32.205857 29658 raft_consensus.cc:482] T 876004fec43e428e930f3fbb873ccac8 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:32.206133 29658 raft_consensus.cc:504] T 876004fec43e428e930f3fbb873ccac8 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:32.206990 29658 leader_election.cc:288] T 876004fec43e428e930f3fbb873ccac8 P 04e36797791842e489aee30de261c4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795), 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815)
I20240502 21:25:32.207785 28755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876004fec43e428e930f3fbb873ccac8" candidate_uuid: "04e36797791842e489aee30de261c4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" is_pre_election: true
I20240502 21:25:32.208096 28755 raft_consensus.cc:2379] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 04e36797791842e489aee30de261c4c7 in current term 1: Already voted for candidate 52c5becf6262415cab0408b3968ed4c0 in this term.
I20240502 21:25:32.207896 29029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876004fec43e428e930f3fbb873ccac8" candidate_uuid: "04e36797791842e489aee30de261c4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c5becf6262415cab0408b3968ed4c0" is_pre_election: true
I20240502 21:25:32.208791 28553 leader_election.cc:302] T 876004fec43e428e930f3fbb873ccac8 P 04e36797791842e489aee30de261c4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 04e36797791842e489aee30de261c4c7; no voters: 52c5becf6262415cab0408b3968ed4c0, 8786a1419d3f4bc79b88c54ce0675aa6
I20240502 21:25:32.209136 29658 raft_consensus.cc:3037] T 876004fec43e428e930f3fbb873ccac8 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:32.211225 29658 raft_consensus.cc:2726] T 876004fec43e428e930f3fbb873ccac8 P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:25:32.337749 29798 consensus_queue.cc:1035] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:32.347684 29799 consensus_queue.cc:1035] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:32.500926 29706 consensus_queue.cc:1035] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:32.501893 29853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:32.502396 29853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 21:25:32.509236 29652 consensus_queue.cc:1035] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:32.515730 29853 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:25:32.524909 29029 raft_consensus.cc:3037] T a604aabc523640089bc54cd6473a9a95 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:32.528225 29776 consensus_queue.cc:1035] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:32.535898 29871 consensus_queue.cc:1035] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'

INSERT report
    rows total: 2000
    time total: 246.924 ms
  time per row: 0.123462 ms
I20240502 21:25:32.954669 28473 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49308:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 21:25:32.967372 28598 tablet_service.cc:1467] Processing CreateTablet for tablet 3665b7b9141b4fe8965eeecea35e9c0b (DEFAULT_TABLE table=rebalance_test_table_3 [id=e6d6bbf7aaec4d548ddf673c06866772]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:32.967690 28735 tablet_service.cc:1467] Processing CreateTablet for tablet 3665b7b9141b4fe8965eeecea35e9c0b (DEFAULT_TABLE table=rebalance_test_table_3 [id=e6d6bbf7aaec4d548ddf673c06866772]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:32.967902 28598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3665b7b9141b4fe8965eeecea35e9c0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:32.967892 28873 tablet_service.cc:1467] Processing CreateTablet for tablet 3665b7b9141b4fe8965eeecea35e9c0b (DEFAULT_TABLE table=rebalance_test_table_3 [id=e6d6bbf7aaec4d548ddf673c06866772]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:32.968456 28735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3665b7b9141b4fe8965eeecea35e9c0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:32.968506 28873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3665b7b9141b4fe8965eeecea35e9c0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:32.971678 29006 tablet_service.cc:1467] Processing CreateTablet for tablet dd7a03f1bc784533bbc10ade78700ba8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e6d6bbf7aaec4d548ddf673c06866772]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:32.972163 29006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd7a03f1bc784533bbc10ade78700ba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:32.979153 29920 tablet_bootstrap.cc:492] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6: Bootstrap starting.
I20240502 21:25:32.981164 29920 tablet_bootstrap.cc:654] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:32.982044 28599 tablet_service.cc:1467] Processing CreateTablet for tablet dd7a03f1bc784533bbc10ade78700ba8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e6d6bbf7aaec4d548ddf673c06866772]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:32.982555 28599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd7a03f1bc784533bbc10ade78700ba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:32.987553 28870 tablet_service.cc:1467] Processing CreateTablet for tablet dd7a03f1bc784533bbc10ade78700ba8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e6d6bbf7aaec4d548ddf673c06866772]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:32.987790 28871 tablet_service.cc:1467] Processing CreateTablet for tablet 2c630e4bed554c8fa99ecfabed6519f1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e6d6bbf7aaec4d548ddf673c06866772]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:32.988018 28870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd7a03f1bc784533bbc10ade78700ba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:32.988338 28871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c630e4bed554c8fa99ecfabed6519f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:32.989460 29007 tablet_service.cc:1467] Processing CreateTablet for tablet 2c630e4bed554c8fa99ecfabed6519f1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e6d6bbf7aaec4d548ddf673c06866772]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:32.989907 29007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c630e4bed554c8fa99ecfabed6519f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:32.997503 28735 tablet_service.cc:1467] Processing CreateTablet for tablet 2c630e4bed554c8fa99ecfabed6519f1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e6d6bbf7aaec4d548ddf673c06866772]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:32.998310 28735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c630e4bed554c8fa99ecfabed6519f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:33.004842 29919 tablet_bootstrap.cc:492] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967: Bootstrap starting.
I20240502 21:25:33.004202 29922 tablet_bootstrap.cc:492] T dd7a03f1bc784533bbc10ade78700ba8 P 04e36797791842e489aee30de261c4c7: Bootstrap starting.
I20240502 21:25:33.005596 29923 tablet_bootstrap.cc:492] T 2c630e4bed554c8fa99ecfabed6519f1 P 52c5becf6262415cab0408b3968ed4c0: Bootstrap starting.
I20240502 21:25:33.007367 29922 tablet_bootstrap.cc:654] T dd7a03f1bc784533bbc10ade78700ba8 P 04e36797791842e489aee30de261c4c7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:33.010051 29920 tablet_bootstrap.cc:492] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6: No bootstrap required, opened a new log
I20240502 21:25:33.010313 29923 tablet_bootstrap.cc:654] T 2c630e4bed554c8fa99ecfabed6519f1 P 52c5becf6262415cab0408b3968ed4c0: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:33.010316 29920 ts_tablet_manager.cc:1367] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6: Time spent bootstrapping tablet: real 0.031s	user 0.004s	sys 0.014s
I20240502 21:25:33.010345 29919 tablet_bootstrap.cc:654] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:33.011497 29920 raft_consensus.cc:348] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:33.011826 29920 raft_consensus.cc:374] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:33.011948 29920 raft_consensus.cc:729] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8786a1419d3f4bc79b88c54ce0675aa6, State: Initialized, Role: FOLLOWER
I20240502 21:25:33.012235 29920 consensus_queue.cc:260] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:33.012813 29919 tablet_bootstrap.cc:492] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967: No bootstrap required, opened a new log
I20240502 21:25:33.012813 29922 tablet_bootstrap.cc:492] T dd7a03f1bc784533bbc10ade78700ba8 P 04e36797791842e489aee30de261c4c7: No bootstrap required, opened a new log
I20240502 21:25:33.012977 29922 ts_tablet_manager.cc:1367] T dd7a03f1bc784533bbc10ade78700ba8 P 04e36797791842e489aee30de261c4c7: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.002s
I20240502 21:25:33.013029 29919 ts_tablet_manager.cc:1367] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967: Time spent bootstrapping tablet: real 0.008s	user 0.004s	sys 0.000s
I20240502 21:25:33.014425 29920 ts_tablet_manager.cc:1398] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 21:25:33.014317 29922 raft_consensus.cc:348] T dd7a03f1bc784533bbc10ade78700ba8 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:25:33.014626 29922 raft_consensus.cc:374] T dd7a03f1bc784533bbc10ade78700ba8 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:33.014780 29922 raft_consensus.cc:729] T dd7a03f1bc784533bbc10ade78700ba8 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e36797791842e489aee30de261c4c7, State: Initialized, Role: FOLLOWER
I20240502 21:25:33.015089 29922 consensus_queue.cc:260] T dd7a03f1bc784533bbc10ade78700ba8 P 04e36797791842e489aee30de261c4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:25:33.015270 29920 tablet_bootstrap.cc:492] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6: Bootstrap starting.
I20240502 21:25:33.015813 29922 ts_tablet_manager.cc:1398] T dd7a03f1bc784533bbc10ade78700ba8 P 04e36797791842e489aee30de261c4c7: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.003s
I20240502 21:25:33.016295 29922 tablet_bootstrap.cc:492] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7: Bootstrap starting.
I20240502 21:25:33.016463 29923 tablet_bootstrap.cc:492] T 2c630e4bed554c8fa99ecfabed6519f1 P 52c5becf6262415cab0408b3968ed4c0: No bootstrap required, opened a new log
I20240502 21:25:33.016649 29923 ts_tablet_manager.cc:1367] T 2c630e4bed554c8fa99ecfabed6519f1 P 52c5becf6262415cab0408b3968ed4c0: Time spent bootstrapping tablet: real 0.011s	user 0.006s	sys 0.000s
I20240502 21:25:33.017037 29920 tablet_bootstrap.cc:654] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:33.018438 29923 raft_consensus.cc:348] T 2c630e4bed554c8fa99ecfabed6519f1 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:33.018746 29923 raft_consensus.cc:374] T 2c630e4bed554c8fa99ecfabed6519f1 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:33.018862 29923 raft_consensus.cc:729] T 2c630e4bed554c8fa99ecfabed6519f1 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c5becf6262415cab0408b3968ed4c0, State: Initialized, Role: FOLLOWER
I20240502 21:25:33.019174 29923 consensus_queue.cc:260] T 2c630e4bed554c8fa99ecfabed6519f1 P 52c5becf6262415cab0408b3968ed4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:33.020627 29922 tablet_bootstrap.cc:654] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:33.021914 29923 ts_tablet_manager.cc:1398] T 2c630e4bed554c8fa99ecfabed6519f1 P 52c5becf6262415cab0408b3968ed4c0: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 21:25:33.022363 29923 tablet_bootstrap.cc:492] T dd7a03f1bc784533bbc10ade78700ba8 P 52c5becf6262415cab0408b3968ed4c0: Bootstrap starting.
I20240502 21:25:33.024688 29923 tablet_bootstrap.cc:654] T dd7a03f1bc784533bbc10ade78700ba8 P 52c5becf6262415cab0408b3968ed4c0: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:33.032845 29919 raft_consensus.cc:348] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:33.033582 29919 raft_consensus.cc:374] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:33.033788 29919 raft_consensus.cc:729] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8733381e64da4e91a8a54e23b01e8967, State: Initialized, Role: FOLLOWER
I20240502 21:25:33.034366 29919 consensus_queue.cc:260] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:33.035755 29919 ts_tablet_manager.cc:1398] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967: Time spent starting tablet: real 0.022s	user 0.003s	sys 0.000s
I20240502 21:25:33.042539 29919 tablet_bootstrap.cc:492] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967: Bootstrap starting.
I20240502 21:25:33.046886 29919 tablet_bootstrap.cc:654] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:33.048097 29923 tablet_bootstrap.cc:492] T dd7a03f1bc784533bbc10ade78700ba8 P 52c5becf6262415cab0408b3968ed4c0: No bootstrap required, opened a new log
I20240502 21:25:33.048391 29923 ts_tablet_manager.cc:1367] T dd7a03f1bc784533bbc10ade78700ba8 P 52c5becf6262415cab0408b3968ed4c0: Time spent bootstrapping tablet: real 0.026s	user 0.023s	sys 0.000s
I20240502 21:25:33.048602 29920 tablet_bootstrap.cc:492] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6: No bootstrap required, opened a new log
I20240502 21:25:33.048869 29920 ts_tablet_manager.cc:1367] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6: Time spent bootstrapping tablet: real 0.034s	user 0.015s	sys 0.014s
I20240502 21:25:33.049204 29922 tablet_bootstrap.cc:492] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7: No bootstrap required, opened a new log
I20240502 21:25:33.049557 29922 ts_tablet_manager.cc:1367] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7: Time spent bootstrapping tablet: real 0.033s	user 0.011s	sys 0.016s
I20240502 21:25:33.049445 29923 raft_consensus.cc:348] T dd7a03f1bc784533bbc10ade78700ba8 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:25:33.049690 29923 raft_consensus.cc:374] T dd7a03f1bc784533bbc10ade78700ba8 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:33.049842 29923 raft_consensus.cc:729] T dd7a03f1bc784533bbc10ade78700ba8 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c5becf6262415cab0408b3968ed4c0, State: Initialized, Role: FOLLOWER
I20240502 21:25:33.049770 29920 raft_consensus.cc:348] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:33.050076 29920 raft_consensus.cc:374] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:33.050215 29920 raft_consensus.cc:729] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8786a1419d3f4bc79b88c54ce0675aa6, State: Initialized, Role: FOLLOWER
I20240502 21:25:33.050258 29923 consensus_queue.cc:260] T dd7a03f1bc784533bbc10ade78700ba8 P 52c5becf6262415cab0408b3968ed4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:25:33.050559 29920 consensus_queue.cc:260] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:33.050786 29922 raft_consensus.cc:348] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:33.051249 29922 raft_consensus.cc:374] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:33.051443 29922 raft_consensus.cc:729] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e36797791842e489aee30de261c4c7, State: Initialized, Role: FOLLOWER
I20240502 21:25:33.051918 29923 ts_tablet_manager.cc:1398] T dd7a03f1bc784533bbc10ade78700ba8 P 52c5becf6262415cab0408b3968ed4c0: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 21:25:33.051815 29922 consensus_queue.cc:260] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:33.052099 29920 ts_tablet_manager.cc:1398] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240502 21:25:33.052438 29922 ts_tablet_manager.cc:1398] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240502 21:25:33.058305 29919 tablet_bootstrap.cc:492] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967: No bootstrap required, opened a new log
I20240502 21:25:33.058593 29919 ts_tablet_manager.cc:1367] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967: Time spent bootstrapping tablet: real 0.022s	user 0.005s	sys 0.005s
I20240502 21:25:33.059636 29919 raft_consensus.cc:348] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:25:33.059909 29919 raft_consensus.cc:374] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:33.059988 29919 raft_consensus.cc:729] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8733381e64da4e91a8a54e23b01e8967, State: Initialized, Role: FOLLOWER
I20240502 21:25:33.060263 29919 consensus_queue.cc:260] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:25:33.060899 29919 ts_tablet_manager.cc:1398] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:25:33.061215 29919 tablet_bootstrap.cc:492] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967: Bootstrap starting.
I20240502 21:25:33.062778 29919 tablet_bootstrap.cc:654] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:33.064889 29919 tablet_bootstrap.cc:492] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967: No bootstrap required, opened a new log
I20240502 21:25:33.065042 29919 ts_tablet_manager.cc:1367] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 21:25:33.065685 29919 raft_consensus.cc:348] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:33.065857 29919 raft_consensus.cc:374] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:33.065965 29919 raft_consensus.cc:729] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8733381e64da4e91a8a54e23b01e8967, State: Initialized, Role: FOLLOWER
I20240502 21:25:33.066201 29919 consensus_queue.cc:260] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:33.066767 29919 ts_tablet_manager.cc:1398] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 21:25:33.222498 29658 raft_consensus.cc:482] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:33.222761 29658 raft_consensus.cc:504] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:33.223654 29658 leader_election.cc:288] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545), 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815)
I20240502 21:25:33.224404 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3665b7b9141b4fe8965eeecea35e9c0b" candidate_uuid: "04e36797791842e489aee30de261c4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" is_pre_election: true
I20240502 21:25:33.224676 28756 raft_consensus.cc:2454] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04e36797791842e489aee30de261c4c7 in term 0.
I20240502 21:25:33.225231 28552 leader_election.cc:302] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04e36797791842e489aee30de261c4c7, 8786a1419d3f4bc79b88c54ce0675aa6; no voters: 
I20240502 21:25:33.225569 29658 raft_consensus.cc:2781] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:33.225687 29658 raft_consensus.cc:482] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:33.225821 29658 raft_consensus.cc:3037] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:33.228071 29658 raft_consensus.cc:504] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:33.228801 29658 leader_election.cc:288] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7 [CANDIDATE]: Term 1 election: Requested vote from peers 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545), 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815)
I20240502 21:25:33.229598 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3665b7b9141b4fe8965eeecea35e9c0b" candidate_uuid: "04e36797791842e489aee30de261c4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8786a1419d3f4bc79b88c54ce0675aa6"
I20240502 21:25:33.229900 28756 raft_consensus.cc:3037] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:33.231408 28893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3665b7b9141b4fe8965eeecea35e9c0b" candidate_uuid: "04e36797791842e489aee30de261c4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8733381e64da4e91a8a54e23b01e8967" is_pre_election: true
I20240502 21:25:33.231498 28892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3665b7b9141b4fe8965eeecea35e9c0b" candidate_uuid: "04e36797791842e489aee30de261c4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8733381e64da4e91a8a54e23b01e8967"
I20240502 21:25:33.231684 28892 raft_consensus.cc:3037] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:33.233008 28756 raft_consensus.cc:2454] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04e36797791842e489aee30de261c4c7 in term 1.
I20240502 21:25:33.234138 28552 leader_election.cc:302] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04e36797791842e489aee30de261c4c7, 8786a1419d3f4bc79b88c54ce0675aa6; no voters: 
I20240502 21:25:33.234401 28892 raft_consensus.cc:2454] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04e36797791842e489aee30de261c4c7 in term 1.
I20240502 21:25:33.234483 29658 raft_consensus.cc:2781] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:33.235394 29658 raft_consensus.cc:686] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7 [term 1 LEADER]: Becoming Leader. State: Replica: 04e36797791842e489aee30de261c4c7, State: Running, Role: LEADER
I20240502 21:25:33.235661 29658 consensus_queue.cc:237] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:33.239217 28471 catalog_manager.cc:5484] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04e36797791842e489aee30de261c4c7 (127.27.196.65). New cstate: current_term: 1 leader_uuid: "04e36797791842e489aee30de261c4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:33.268338 29706 raft_consensus.cc:482] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:33.268661 29706 raft_consensus.cc:504] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:33.269416 29706 leader_election.cc:288] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795), 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815)
I20240502 21:25:33.269939 29030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c630e4bed554c8fa99ecfabed6519f1" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c5becf6262415cab0408b3968ed4c0" is_pre_election: true
I20240502 21:25:33.269991 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c630e4bed554c8fa99ecfabed6519f1" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" is_pre_election: true
I20240502 21:25:33.270195 29030 raft_consensus.cc:2454] T 2c630e4bed554c8fa99ecfabed6519f1 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8733381e64da4e91a8a54e23b01e8967 in term 0.
I20240502 21:25:33.270224 28756 raft_consensus.cc:2454] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8733381e64da4e91a8a54e23b01e8967 in term 0.
I20240502 21:25:33.270671 28825 leader_election.cc:302] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52c5becf6262415cab0408b3968ed4c0, 8733381e64da4e91a8a54e23b01e8967; no voters: 
I20240502 21:25:33.270992 29706 raft_consensus.cc:2781] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:33.271143 29706 raft_consensus.cc:482] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:33.271243 29706 raft_consensus.cc:3037] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:33.273231 29706 raft_consensus.cc:504] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:33.273923 29706 leader_election.cc:288] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 election: Requested vote from peers 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795), 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815)
I20240502 21:25:33.274727 29030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c630e4bed554c8fa99ecfabed6519f1" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c5becf6262415cab0408b3968ed4c0"
I20240502 21:25:33.275012 29030 raft_consensus.cc:3037] T 2c630e4bed554c8fa99ecfabed6519f1 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:33.275096 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c630e4bed554c8fa99ecfabed6519f1" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8786a1419d3f4bc79b88c54ce0675aa6"
I20240502 21:25:33.275504 28756 raft_consensus.cc:3037] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:33.277227 29030 raft_consensus.cc:2454] T 2c630e4bed554c8fa99ecfabed6519f1 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8733381e64da4e91a8a54e23b01e8967 in term 1.
I20240502 21:25:33.277823 28825 leader_election.cc:302] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52c5becf6262415cab0408b3968ed4c0, 8733381e64da4e91a8a54e23b01e8967; no voters: 
I20240502 21:25:33.277884 28756 raft_consensus.cc:2454] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8733381e64da4e91a8a54e23b01e8967 in term 1.
I20240502 21:25:33.278172 29706 raft_consensus.cc:2781] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:33.278628 29706 raft_consensus.cc:686] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Becoming Leader. State: Replica: 8733381e64da4e91a8a54e23b01e8967, State: Running, Role: LEADER
I20240502 21:25:33.278990 29706 consensus_queue.cc:237] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:33.282806 28471 catalog_manager.cc:5484] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67). New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:33.321090 29936 raft_consensus.cc:482] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:33.321316 29936 raft_consensus.cc:504] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:25:33.322400 29936 leader_election.cc:288] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795), 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983)
I20240502 21:25:33.323019 28620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7a03f1bc784533bbc10ade78700ba8" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04e36797791842e489aee30de261c4c7" is_pre_election: true
I20240502 21:25:33.323024 29030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7a03f1bc784533bbc10ade78700ba8" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c5becf6262415cab0408b3968ed4c0" is_pre_election: true
I20240502 21:25:33.323307 29030 raft_consensus.cc:2454] T dd7a03f1bc784533bbc10ade78700ba8 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8733381e64da4e91a8a54e23b01e8967 in term 0.
I20240502 21:25:33.323307 28620 raft_consensus.cc:2454] T dd7a03f1bc784533bbc10ade78700ba8 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8733381e64da4e91a8a54e23b01e8967 in term 0.
I20240502 21:25:33.323773 28825 leader_election.cc:302] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52c5becf6262415cab0408b3968ed4c0, 8733381e64da4e91a8a54e23b01e8967; no voters: 
I20240502 21:25:33.324083 29936 raft_consensus.cc:2781] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:33.324187 29936 raft_consensus.cc:482] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:33.324280 29936 raft_consensus.cc:3037] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:33.327078 29936 raft_consensus.cc:504] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:25:33.328007 29936 leader_election.cc:288] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 election: Requested vote from peers 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795), 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983)
I20240502 21:25:33.328622 28620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7a03f1bc784533bbc10ade78700ba8" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04e36797791842e489aee30de261c4c7"
I20240502 21:25:33.328652 29030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7a03f1bc784533bbc10ade78700ba8" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c5becf6262415cab0408b3968ed4c0"
I20240502 21:25:33.328840 29030 raft_consensus.cc:3037] T dd7a03f1bc784533bbc10ade78700ba8 P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:33.328864 28620 raft_consensus.cc:3037] T dd7a03f1bc784533bbc10ade78700ba8 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:33.331060 28620 raft_consensus.cc:2454] T dd7a03f1bc784533bbc10ade78700ba8 P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8733381e64da4e91a8a54e23b01e8967 in term 1.
I20240502 21:25:33.331146 29030 raft_consensus.cc:2454] T dd7a03f1bc784533bbc10ade78700ba8 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8733381e64da4e91a8a54e23b01e8967 in term 1.
I20240502 21:25:33.331646 28824 leader_election.cc:302] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04e36797791842e489aee30de261c4c7, 8733381e64da4e91a8a54e23b01e8967; no voters: 
I20240502 21:25:33.332103 29936 raft_consensus.cc:2781] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:33.332273 29936 raft_consensus.cc:686] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Becoming Leader. State: Replica: 8733381e64da4e91a8a54e23b01e8967, State: Running, Role: LEADER
I20240502 21:25:33.332569 29936 consensus_queue.cc:237] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:25:33.335748 28471 catalog_manager.cc:5484] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67). New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:33.732471 29658 consensus_queue.cc:1035] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:33.738806 29943 consensus_queue.cc:1035] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:33.748241 29939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:33.748627 29939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:33.756873 29939 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:25:33.763844 29936 consensus_queue.cc:1035] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:33.770404 29936 consensus_queue.cc:1035] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:33.790565 29950 consensus_queue.cc:1035] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:33.797941 29706 consensus_queue.cc:1035] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'

INSERT report
    rows total: 2000
    time total: 269.45 ms
  time per row: 0.134725 ms
I20240502 21:25:34.211007 28471 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49308:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 21:25:34.224885 28735 tablet_service.cc:1467] Processing CreateTablet for tablet a4c6f9c73a2d4b01a2c027504e45b238 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9474d8b6526a4d809c3b7e67f807f596]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:34.224896 28598 tablet_service.cc:1467] Processing CreateTablet for tablet a4c6f9c73a2d4b01a2c027504e45b238 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9474d8b6526a4d809c3b7e67f807f596]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:34.225548 28735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c6f9c73a2d4b01a2c027504e45b238. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:34.225561 28598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c6f9c73a2d4b01a2c027504e45b238. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:34.228446 29007 tablet_service.cc:1467] Processing CreateTablet for tablet a2fd105773d449da89940d3b6558361a (DEFAULT_TABLE table=rebalance_test_table_4 [id=9474d8b6526a4d809c3b7e67f807f596]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:34.229080 29007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2fd105773d449da89940d3b6558361a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:34.229175 28873 tablet_service.cc:1467] Processing CreateTablet for tablet a4c6f9c73a2d4b01a2c027504e45b238 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9474d8b6526a4d809c3b7e67f807f596]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:34.230031 28873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c6f9c73a2d4b01a2c027504e45b238. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:34.231029 28736 tablet_service.cc:1467] Processing CreateTablet for tablet a2fd105773d449da89940d3b6558361a (DEFAULT_TABLE table=rebalance_test_table_4 [id=9474d8b6526a4d809c3b7e67f807f596]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:34.231629 28736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2fd105773d449da89940d3b6558361a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:34.231681 29006 tablet_service.cc:1467] Processing CreateTablet for tablet e6387ed203114433b587600b1a8a682a (DEFAULT_TABLE table=rebalance_test_table_4 [id=9474d8b6526a4d809c3b7e67f807f596]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:34.232152 29006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6387ed203114433b587600b1a8a682a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:34.232245 28599 tablet_service.cc:1467] Processing CreateTablet for tablet a2fd105773d449da89940d3b6558361a (DEFAULT_TABLE table=rebalance_test_table_4 [id=9474d8b6526a4d809c3b7e67f807f596]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:34.232707 28599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2fd105773d449da89940d3b6558361a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:34.235594 28597 tablet_service.cc:1467] Processing CreateTablet for tablet e6387ed203114433b587600b1a8a682a (DEFAULT_TABLE table=rebalance_test_table_4 [id=9474d8b6526a4d809c3b7e67f807f596]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:34.236038 28597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6387ed203114433b587600b1a8a682a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:34.235976 28871 tablet_service.cc:1467] Processing CreateTablet for tablet e6387ed203114433b587600b1a8a682a (DEFAULT_TABLE table=rebalance_test_table_4 [id=9474d8b6526a4d809c3b7e67f807f596]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:34.236680 28871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6387ed203114433b587600b1a8a682a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:34.257791 30007 tablet_bootstrap.cc:492] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6: Bootstrap starting.
I20240502 21:25:34.260944 30007 tablet_bootstrap.cc:654] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:34.262697 30008 tablet_bootstrap.cc:492] T e6387ed203114433b587600b1a8a682a P 04e36797791842e489aee30de261c4c7: Bootstrap starting.
I20240502 21:25:34.264161 30010 tablet_bootstrap.cc:492] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967: Bootstrap starting.
I20240502 21:25:34.264935 30008 tablet_bootstrap.cc:654] T e6387ed203114433b587600b1a8a682a P 04e36797791842e489aee30de261c4c7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:34.266304 30009 tablet_bootstrap.cc:492] T a2fd105773d449da89940d3b6558361a P 52c5becf6262415cab0408b3968ed4c0: Bootstrap starting.
I20240502 21:25:34.266916 30010 tablet_bootstrap.cc:654] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:34.268148 30009 tablet_bootstrap.cc:654] T a2fd105773d449da89940d3b6558361a P 52c5becf6262415cab0408b3968ed4c0: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:34.268556 30008 tablet_bootstrap.cc:492] T e6387ed203114433b587600b1a8a682a P 04e36797791842e489aee30de261c4c7: No bootstrap required, opened a new log
I20240502 21:25:34.268777 30008 ts_tablet_manager.cc:1367] T e6387ed203114433b587600b1a8a682a P 04e36797791842e489aee30de261c4c7: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.004s
I20240502 21:25:34.272706 30008 raft_consensus.cc:348] T e6387ed203114433b587600b1a8a682a P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:34.273223 30008 raft_consensus.cc:374] T e6387ed203114433b587600b1a8a682a P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:34.273816 30008 raft_consensus.cc:729] T e6387ed203114433b587600b1a8a682a P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e36797791842e489aee30de261c4c7, State: Initialized, Role: FOLLOWER
I20240502 21:25:34.274219 30008 consensus_queue.cc:260] T e6387ed203114433b587600b1a8a682a P 04e36797791842e489aee30de261c4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:34.275496 30008 ts_tablet_manager.cc:1398] T e6387ed203114433b587600b1a8a682a P 04e36797791842e489aee30de261c4c7: Time spent starting tablet: real 0.007s	user 0.000s	sys 0.003s
I20240502 21:25:34.276408 30008 tablet_bootstrap.cc:492] T a4c6f9c73a2d4b01a2c027504e45b238 P 04e36797791842e489aee30de261c4c7: Bootstrap starting.
I20240502 21:25:34.278506 30009 tablet_bootstrap.cc:492] T a2fd105773d449da89940d3b6558361a P 52c5becf6262415cab0408b3968ed4c0: No bootstrap required, opened a new log
I20240502 21:25:34.278826 30009 ts_tablet_manager.cc:1367] T a2fd105773d449da89940d3b6558361a P 52c5becf6262415cab0408b3968ed4c0: Time spent bootstrapping tablet: real 0.013s	user 0.010s	sys 0.000s
I20240502 21:25:34.279506 30007 tablet_bootstrap.cc:492] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6: No bootstrap required, opened a new log
I20240502 21:25:34.279568 30008 tablet_bootstrap.cc:654] T a4c6f9c73a2d4b01a2c027504e45b238 P 04e36797791842e489aee30de261c4c7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:34.279704 30007 ts_tablet_manager.cc:1367] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6: Time spent bootstrapping tablet: real 0.022s	user 0.006s	sys 0.011s
I20240502 21:25:34.279939 30009 raft_consensus.cc:348] T a2fd105773d449da89940d3b6558361a P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } }
I20240502 21:25:34.280269 30009 raft_consensus.cc:374] T a2fd105773d449da89940d3b6558361a P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:34.280433 30009 raft_consensus.cc:729] T a2fd105773d449da89940d3b6558361a P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c5becf6262415cab0408b3968ed4c0, State: Initialized, Role: FOLLOWER
I20240502 21:25:34.280675 30007 raft_consensus.cc:348] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:34.280987 30007 raft_consensus.cc:374] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:34.280805 30009 consensus_queue.cc:260] T a2fd105773d449da89940d3b6558361a P 52c5becf6262415cab0408b3968ed4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } }
I20240502 21:25:34.281105 30007 raft_consensus.cc:729] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8786a1419d3f4bc79b88c54ce0675aa6, State: Initialized, Role: FOLLOWER
I20240502 21:25:34.281914 30009 ts_tablet_manager.cc:1398] T a2fd105773d449da89940d3b6558361a P 52c5becf6262415cab0408b3968ed4c0: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 21:25:34.282331 30009 tablet_bootstrap.cc:492] T e6387ed203114433b587600b1a8a682a P 52c5becf6262415cab0408b3968ed4c0: Bootstrap starting.
I20240502 21:25:34.281415 30007 consensus_queue.cc:260] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:34.284291 30009 tablet_bootstrap.cc:654] T e6387ed203114433b587600b1a8a682a P 52c5becf6262415cab0408b3968ed4c0: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:34.285166 30007 ts_tablet_manager.cc:1398] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 21:25:34.285579 30007 tablet_bootstrap.cc:492] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6: Bootstrap starting.
I20240502 21:25:34.289158 30010 tablet_bootstrap.cc:492] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967: No bootstrap required, opened a new log
I20240502 21:25:34.289399 30010 ts_tablet_manager.cc:1367] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967: Time spent bootstrapping tablet: real 0.025s	user 0.008s	sys 0.000s
I20240502 21:25:34.289800 30008 tablet_bootstrap.cc:492] T a4c6f9c73a2d4b01a2c027504e45b238 P 04e36797791842e489aee30de261c4c7: No bootstrap required, opened a new log
I20240502 21:25:34.290006 30008 ts_tablet_manager.cc:1367] T a4c6f9c73a2d4b01a2c027504e45b238 P 04e36797791842e489aee30de261c4c7: Time spent bootstrapping tablet: real 0.014s	user 0.006s	sys 0.005s
I20240502 21:25:34.290316 30010 raft_consensus.cc:348] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:34.290614 30010 raft_consensus.cc:374] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:34.290730 30010 raft_consensus.cc:729] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8733381e64da4e91a8a54e23b01e8967, State: Initialized, Role: FOLLOWER
I20240502 21:25:34.290918 30008 raft_consensus.cc:348] T a4c6f9c73a2d4b01a2c027504e45b238 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:34.291172 30008 raft_consensus.cc:374] T a4c6f9c73a2d4b01a2c027504e45b238 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:34.291265 30008 raft_consensus.cc:729] T a4c6f9c73a2d4b01a2c027504e45b238 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e36797791842e489aee30de261c4c7, State: Initialized, Role: FOLLOWER
I20240502 21:25:34.291066 30010 consensus_queue.cc:260] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:34.291602 30008 consensus_queue.cc:260] T a4c6f9c73a2d4b01a2c027504e45b238 P 04e36797791842e489aee30de261c4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:34.292016 30010 ts_tablet_manager.cc:1398] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:25:34.292197 30008 ts_tablet_manager.cc:1398] T a4c6f9c73a2d4b01a2c027504e45b238 P 04e36797791842e489aee30de261c4c7: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 21:25:34.292310 30010 tablet_bootstrap.cc:492] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967: Bootstrap starting.
I20240502 21:25:34.294085 30010 tablet_bootstrap.cc:654] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:34.294164 30009 tablet_bootstrap.cc:492] T e6387ed203114433b587600b1a8a682a P 52c5becf6262415cab0408b3968ed4c0: No bootstrap required, opened a new log
I20240502 21:25:34.294373 30009 ts_tablet_manager.cc:1367] T e6387ed203114433b587600b1a8a682a P 52c5becf6262415cab0408b3968ed4c0: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.003s
I20240502 21:25:34.294417 30007 tablet_bootstrap.cc:654] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:34.295460 30009 raft_consensus.cc:348] T e6387ed203114433b587600b1a8a682a P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:34.295715 30009 raft_consensus.cc:374] T e6387ed203114433b587600b1a8a682a P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:34.295805 30009 raft_consensus.cc:729] T e6387ed203114433b587600b1a8a682a P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c5becf6262415cab0408b3968ed4c0, State: Initialized, Role: FOLLOWER
I20240502 21:25:34.296144 30009 consensus_queue.cc:260] T e6387ed203114433b587600b1a8a682a P 52c5becf6262415cab0408b3968ed4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:34.296885 30009 ts_tablet_manager.cc:1398] T e6387ed203114433b587600b1a8a682a P 52c5becf6262415cab0408b3968ed4c0: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 21:25:34.296993 30008 tablet_bootstrap.cc:492] T a2fd105773d449da89940d3b6558361a P 04e36797791842e489aee30de261c4c7: Bootstrap starting.
I20240502 21:25:34.299638 30007 tablet_bootstrap.cc:492] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6: No bootstrap required, opened a new log
I20240502 21:25:34.299912 30007 ts_tablet_manager.cc:1367] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6: Time spent bootstrapping tablet: real 0.014s	user 0.003s	sys 0.001s
I20240502 21:25:34.300460 30008 tablet_bootstrap.cc:654] T a2fd105773d449da89940d3b6558361a P 04e36797791842e489aee30de261c4c7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:34.300474 30010 tablet_bootstrap.cc:492] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967: No bootstrap required, opened a new log
I20240502 21:25:34.300653 30010 ts_tablet_manager.cc:1367] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967: Time spent bootstrapping tablet: real 0.008s	user 0.006s	sys 0.000s
I20240502 21:25:34.300940 30007 raft_consensus.cc:348] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } }
I20240502 21:25:34.301249 30007 raft_consensus.cc:374] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:34.301359 30007 raft_consensus.cc:729] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8786a1419d3f4bc79b88c54ce0675aa6, State: Initialized, Role: FOLLOWER
I20240502 21:25:34.301662 30007 consensus_queue.cc:260] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } }
I20240502 21:25:34.301774 30010 raft_consensus.cc:348] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:34.302035 30010 raft_consensus.cc:374] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:34.302170 30010 raft_consensus.cc:729] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8733381e64da4e91a8a54e23b01e8967, State: Initialized, Role: FOLLOWER
I20240502 21:25:34.302457 30007 ts_tablet_manager.cc:1398] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:25:34.302650 30008 tablet_bootstrap.cc:492] T a2fd105773d449da89940d3b6558361a P 04e36797791842e489aee30de261c4c7: No bootstrap required, opened a new log
I20240502 21:25:34.302529 30010 consensus_queue.cc:260] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:34.302878 30008 ts_tablet_manager.cc:1367] T a2fd105773d449da89940d3b6558361a P 04e36797791842e489aee30de261c4c7: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240502 21:25:34.303470 30010 ts_tablet_manager.cc:1398] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 21:25:34.303767 30008 raft_consensus.cc:348] T a2fd105773d449da89940d3b6558361a P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } }
I20240502 21:25:34.304004 30008 raft_consensus.cc:374] T a2fd105773d449da89940d3b6558361a P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:34.304157 30008 raft_consensus.cc:729] T a2fd105773d449da89940d3b6558361a P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e36797791842e489aee30de261c4c7, State: Initialized, Role: FOLLOWER
I20240502 21:25:34.304463 30008 consensus_queue.cc:260] T a2fd105773d449da89940d3b6558361a P 04e36797791842e489aee30de261c4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } }
I20240502 21:25:34.305102 30008 ts_tablet_manager.cc:1398] T a2fd105773d449da89940d3b6558361a P 04e36797791842e489aee30de261c4c7: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240502 21:25:34.397393 29950 raft_consensus.cc:482] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:34.397738 29950 raft_consensus.cc:504] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:34.398774 29950 leader_election.cc:288] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815), 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983)
I20240502 21:25:34.399878 28619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c6f9c73a2d4b01a2c027504e45b238" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04e36797791842e489aee30de261c4c7" is_pre_election: true
I20240502 21:25:34.399878 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c6f9c73a2d4b01a2c027504e45b238" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" is_pre_election: true
I20240502 21:25:34.400211 28756 raft_consensus.cc:2454] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8733381e64da4e91a8a54e23b01e8967 in term 0.
I20240502 21:25:34.400205 28619 raft_consensus.cc:2454] T a4c6f9c73a2d4b01a2c027504e45b238 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8733381e64da4e91a8a54e23b01e8967 in term 0.
I20240502 21:25:34.400830 28824 leader_election.cc:302] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8733381e64da4e91a8a54e23b01e8967, 8786a1419d3f4bc79b88c54ce0675aa6; no voters: 
I20240502 21:25:34.401199 29950 raft_consensus.cc:2781] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:34.401319 29950 raft_consensus.cc:482] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:34.401434 29950 raft_consensus.cc:3037] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:34.403843 29950 raft_consensus.cc:504] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:34.404683 29950 leader_election.cc:288] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 election: Requested vote from peers 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815), 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983)
I20240502 21:25:34.405381 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c6f9c73a2d4b01a2c027504e45b238" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8786a1419d3f4bc79b88c54ce0675aa6"
I20240502 21:25:34.405489 28619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c6f9c73a2d4b01a2c027504e45b238" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04e36797791842e489aee30de261c4c7"
I20240502 21:25:34.405637 28756 raft_consensus.cc:3037] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:34.405819 28619 raft_consensus.cc:3037] T a4c6f9c73a2d4b01a2c027504e45b238 P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:34.408142 28619 raft_consensus.cc:2454] T a4c6f9c73a2d4b01a2c027504e45b238 P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8733381e64da4e91a8a54e23b01e8967 in term 1.
I20240502 21:25:34.408170 28756 raft_consensus.cc:2454] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8733381e64da4e91a8a54e23b01e8967 in term 1.
I20240502 21:25:34.408739 28824 leader_election.cc:302] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8733381e64da4e91a8a54e23b01e8967, 8786a1419d3f4bc79b88c54ce0675aa6; no voters: 
I20240502 21:25:34.409121 29950 raft_consensus.cc:2781] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:34.409343 29950 raft_consensus.cc:686] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Becoming Leader. State: Replica: 8733381e64da4e91a8a54e23b01e8967, State: Running, Role: LEADER
I20240502 21:25:34.409657 29950 consensus_queue.cc:237] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:34.413668 28474 catalog_manager.cc:5484] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67). New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:34.428316 29950 raft_consensus.cc:482] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:34.428510 29950 raft_consensus.cc:504] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:34.429214 29950 leader_election.cc:288] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983), 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795)
I20240502 21:25:34.429777 29030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6387ed203114433b587600b1a8a682a" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c5becf6262415cab0408b3968ed4c0" is_pre_election: true
I20240502 21:25:34.429778 28619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6387ed203114433b587600b1a8a682a" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04e36797791842e489aee30de261c4c7" is_pre_election: true
I20240502 21:25:34.430073 29030 raft_consensus.cc:2454] T e6387ed203114433b587600b1a8a682a P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8733381e64da4e91a8a54e23b01e8967 in term 0.
I20240502 21:25:34.430073 28619 raft_consensus.cc:2454] T e6387ed203114433b587600b1a8a682a P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8733381e64da4e91a8a54e23b01e8967 in term 0.
I20240502 21:25:34.430538 28825 leader_election.cc:302] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52c5becf6262415cab0408b3968ed4c0, 8733381e64da4e91a8a54e23b01e8967; no voters: 
I20240502 21:25:34.430846 29950 raft_consensus.cc:2781] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:34.430980 29950 raft_consensus.cc:482] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:34.431084 29950 raft_consensus.cc:3037] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:34.433274 29950 raft_consensus.cc:504] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:34.433925 29950 leader_election.cc:288] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 election: Requested vote from peers 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983), 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795)
I20240502 21:25:34.434504 28619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6387ed203114433b587600b1a8a682a" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04e36797791842e489aee30de261c4c7"
I20240502 21:25:34.434741 28619 raft_consensus.cc:3037] T e6387ed203114433b587600b1a8a682a P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:34.434855 29030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6387ed203114433b587600b1a8a682a" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c5becf6262415cab0408b3968ed4c0"
I20240502 21:25:34.435256 29030 raft_consensus.cc:3037] T e6387ed203114433b587600b1a8a682a P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:34.437244 28619 raft_consensus.cc:2454] T e6387ed203114433b587600b1a8a682a P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8733381e64da4e91a8a54e23b01e8967 in term 1.
I20240502 21:25:34.437875 28824 leader_election.cc:302] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04e36797791842e489aee30de261c4c7, 8733381e64da4e91a8a54e23b01e8967; no voters: 
I20240502 21:25:34.438249 29950 raft_consensus.cc:2781] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:34.438266 29030 raft_consensus.cc:2454] T e6387ed203114433b587600b1a8a682a P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8733381e64da4e91a8a54e23b01e8967 in term 1.
I20240502 21:25:34.438475 29950 raft_consensus.cc:686] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Becoming Leader. State: Replica: 8733381e64da4e91a8a54e23b01e8967, State: Running, Role: LEADER
I20240502 21:25:34.439306 29950 consensus_queue.cc:237] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:34.443202 28474 catalog_manager.cc:5484] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67). New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:34.660540 29870 raft_consensus.cc:482] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:34.660853 29870 raft_consensus.cc:504] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } }
I20240502 21:25:34.661888 29870 leader_election.cc:288] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983), 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795)
I20240502 21:25:34.662706 28619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2fd105773d449da89940d3b6558361a" candidate_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04e36797791842e489aee30de261c4c7" is_pre_election: true
I20240502 21:25:34.663000 28619 raft_consensus.cc:2454] T a2fd105773d449da89940d3b6558361a P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8786a1419d3f4bc79b88c54ce0675aa6 in term 0.
I20240502 21:25:34.663025 29030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2fd105773d449da89940d3b6558361a" candidate_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c5becf6262415cab0408b3968ed4c0" is_pre_election: true
I20240502 21:25:34.663344 29030 raft_consensus.cc:2454] T a2fd105773d449da89940d3b6558361a P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8786a1419d3f4bc79b88c54ce0675aa6 in term 0.
I20240502 21:25:34.663586 28687 leader_election.cc:302] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04e36797791842e489aee30de261c4c7, 8786a1419d3f4bc79b88c54ce0675aa6; no voters: 
I20240502 21:25:34.663942 29870 raft_consensus.cc:2781] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:34.664089 29870 raft_consensus.cc:482] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:34.664211 29870 raft_consensus.cc:3037] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:34.666543 29870 raft_consensus.cc:504] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } }
I20240502 21:25:34.667409 29870 leader_election.cc:288] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 1 election: Requested vote from peers 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983), 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795)
I20240502 21:25:34.668272 28619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2fd105773d449da89940d3b6558361a" candidate_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04e36797791842e489aee30de261c4c7"
I20240502 21:25:34.668311 29030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2fd105773d449da89940d3b6558361a" candidate_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c5becf6262415cab0408b3968ed4c0"
I20240502 21:25:34.668557 28619 raft_consensus.cc:3037] T a2fd105773d449da89940d3b6558361a P 04e36797791842e489aee30de261c4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:34.668581 29030 raft_consensus.cc:3037] T a2fd105773d449da89940d3b6558361a P 52c5becf6262415cab0408b3968ed4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:34.670614 28619 raft_consensus.cc:2454] T a2fd105773d449da89940d3b6558361a P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8786a1419d3f4bc79b88c54ce0675aa6 in term 1.
I20240502 21:25:34.670677 29030 raft_consensus.cc:2454] T a2fd105773d449da89940d3b6558361a P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8786a1419d3f4bc79b88c54ce0675aa6 in term 1.
I20240502 21:25:34.671281 28688 leader_election.cc:302] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52c5becf6262415cab0408b3968ed4c0, 8786a1419d3f4bc79b88c54ce0675aa6; no voters: 
I20240502 21:25:34.671690 29870 raft_consensus.cc:2781] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:34.672087 29870 raft_consensus.cc:686] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 LEADER]: Becoming Leader. State: Replica: 8786a1419d3f4bc79b88c54ce0675aa6, State: Running, Role: LEADER
I20240502 21:25:34.672690 29870 consensus_queue.cc:237] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } }
I20240502 21:25:34.675832 28474 catalog_manager.cc:5484] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:34.853369 29950 consensus_queue.cc:1035] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:34.866499 29706 consensus_queue.cc:1035] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:34.927726 29706 consensus_queue.cc:1035] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:34.937161 29950 consensus_queue.cc:1035] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:35.090894 29870 consensus_queue.cc:1035] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:35.099002 30035 consensus_queue.cc:1035] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:35.204253 30025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:35.204696 30025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:35.213697 30025 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'

INSERT report
    rows total: 2000
    time total: 307.916 ms
  time per row: 0.153958 ms
I20240502 21:25:35.696180 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.69:41545
--local_ip_for_outbound_sockets=127.27.196.69
--tserver_master_addrs=127.27.196.126:39011
--webserver_port=33279
--webserver_interface=127.27.196.69
--builtin_ntp_servers=127.27.196.84:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 21:25:36.238307 30087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:36.238719 30087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:36.238926 30087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:36.247367 30087 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:36.247501 30087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:36.247665 30087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.69
I20240502 21:25:36.258020 30087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.69:41545
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.27.196.69
--webserver_port=33279
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.27.196.126:39011
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.69
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:36.259557 30087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:36.261402 30087 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:36.275029 30093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:36.275790 30096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:36.276340 30094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:36.276824 30087 server_base.cc:1034] running on GCE node
I20240502 21:25:36.277623 30087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:36.278672 30087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:36.279903 30087 hybrid_clock.cc:648] HybridClock initialized: now 1714685136279859 us; error 39 us; skew 500 ppm
I20240502 21:25:36.280267 30087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:36.283988 30087 webserver.cc:460] Webserver started at http://127.27.196.69:33279/ using document root <none> and password file <none>
I20240502 21:25:36.285216 30087 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:36.285413 30087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:36.293917 30087 fs_manager.cc:709] Time spent opening directory manager: real 0.006s	user 0.001s	sys 0.003s
I20240502 21:25:36.298736 30103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:36.301010 30087 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.002s	sys 0.000s
I20240502 21:25:36.301306 30087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "a1e2dba8e1a3403295cb394f7090f849"
format_stamp: "Formatted at 2024-05-02 21:25:23 on dist-test-slave-3bs1"
I20240502 21:25:36.302484 30087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:36.329996 30087 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:25:36.331027 30087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:36.331400 30087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:36.332542 30087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:36.334219 30087 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:36.334298 30087 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:36.334398 30087 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:36.334478 30087 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:36.379701 30087 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.69:41545
I20240502 21:25:36.379724 30216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.69:41545 every 8 connection(s)
I20240502 21:25:36.381425 30087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:25:36.389819 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 30087
I20240502 21:25:36.390703 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.70:43969
--local_ip_for_outbound_sockets=127.27.196.70
--tserver_master_addrs=127.27.196.126:39011
--webserver_port=38885
--webserver_interface=127.27.196.70
--builtin_ntp_servers=127.27.196.84:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 21:25:36.398376 30217 heartbeater.cc:347] Connected to a master server at 127.27.196.126:39011
I20240502 21:25:36.398780 30217 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:36.399957 30217 heartbeater.cc:511] Master 127.27.196.126:39011 requested a full tablet report, sending...
I20240502 21:25:36.401713 28476 ts_manager.cc:194] Re-registered known tserver with Master: a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69:41545)
I20240502 21:25:36.402632 28476 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.69:58885
W20240502 21:25:36.785171 30222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:36.785625 30222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:36.785809 30222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:36.794152 30222 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:36.794349 30222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:36.794588 30222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.70
I20240502 21:25:36.804729 30222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.70:43969
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.27.196.70
--webserver_port=38885
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.27.196.126:39011
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.70
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:36.806233 30222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:36.808498 30222 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:36.823251 30229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:36.823529 30228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:36.824031 30231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:36.825125 30222 server_base.cc:1034] running on GCE node
I20240502 21:25:36.826066 30222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:36.827447 30222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:36.828771 30222 hybrid_clock.cc:648] HybridClock initialized: now 1714685136828696 us; error 74 us; skew 500 ppm
I20240502 21:25:36.829175 30222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:36.832636 30222 webserver.cc:460] Webserver started at http://127.27.196.70:38885/ using document root <none> and password file <none>
I20240502 21:25:36.833788 30222 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:36.833950 30222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:36.841667 30222 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240502 21:25:36.845860 30237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:36.848073 30222 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.002s	sys 0.001s
I20240502 21:25:36.848362 30222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "fc5aa2dc1f7b4554906785d693e272d0"
format_stamp: "Formatted at 2024-05-02 21:25:23 on dist-test-slave-3bs1"
I20240502 21:25:36.849551 30222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:36.881634 30222 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:25:36.882643 30222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:36.882989 30222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:36.884182 30222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:36.886286 30222 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:36.886379 30222 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:36.886497 30222 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:36.886555 30222 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:36.929670 30222 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.70:43969
I20240502 21:25:36.929728 30352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.70:43969 every 8 connection(s)
I20240502 21:25:36.931536 30222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 21:25:36.937530 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 30222
I20240502 21:25:36.938419 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.71:36827
--local_ip_for_outbound_sockets=127.27.196.71
--tserver_master_addrs=127.27.196.126:39011
--webserver_port=44401
--webserver_interface=127.27.196.71
--builtin_ntp_servers=127.27.196.84:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 21:25:36.951712 30353 heartbeater.cc:347] Connected to a master server at 127.27.196.126:39011
I20240502 21:25:36.952121 30353 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:36.953119 30353 heartbeater.cc:511] Master 127.27.196.126:39011 requested a full tablet report, sending...
I20240502 21:25:36.954936 28476 ts_manager.cc:194] Re-registered known tserver with Master: fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70:43969)
I20240502 21:25:36.955894 28476 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.70:60047
I20240502 21:25:37.405449 30217 heartbeater.cc:503] Master 127.27.196.126:39011 was elected leader, sending a full tablet report...
W20240502 21:25:37.471426 30357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:37.471889 30357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:37.472057 30357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:37.482031 30357 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:37.482270 30357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:37.482517 30357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.71
I20240502 21:25:37.493659 30357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.71:36827
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.27.196.71
--webserver_port=44401
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.27.196.126:39011
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.71
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:37.495354 30357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:37.497483 30357 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:37.512367 30364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:37.512791 30366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:37.512837 30363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:37.513868 30357 server_base.cc:1034] running on GCE node
I20240502 21:25:37.514847 30357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:37.516156 30357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:37.517486 30357 hybrid_clock.cc:648] HybridClock initialized: now 1714685137517395 us; error 43 us; skew 500 ppm
I20240502 21:25:37.518054 30357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:37.521528 30357 webserver.cc:460] Webserver started at http://127.27.196.71:44401/ using document root <none> and password file <none>
I20240502 21:25:37.522743 30357 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:37.522914 30357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:37.530213 30357 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.001s	sys 0.004s
I20240502 21:25:37.533998 30372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:37.535954 30357 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.002s
I20240502 21:25:37.536175 30357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "a9ca254f89984b3f9ed62020bd6a4d02"
format_stamp: "Formatted at 2024-05-02 21:25:24 on dist-test-slave-3bs1"
I20240502 21:25:37.537200 30357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:37.575562 30357 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:25:37.576593 30357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:37.576926 30357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:37.578155 30357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:37.580077 30357 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:37.580166 30357 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:37.580312 30357 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:37.580372 30357 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:37.623435 30357 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.71:36827
I20240502 21:25:37.623494 30485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.71:36827 every 8 connection(s)
I20240502 21:25:37.625272 30357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 21:25:37.630398 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 30357
I20240502 21:25:37.631757 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.72:34997
--local_ip_for_outbound_sockets=127.27.196.72
--tserver_master_addrs=127.27.196.126:39011
--webserver_port=35791
--webserver_interface=127.27.196.72
--builtin_ntp_servers=127.27.196.84:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 21:25:37.644955 30486 heartbeater.cc:347] Connected to a master server at 127.27.196.126:39011
I20240502 21:25:37.645372 30486 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:37.646574 30486 heartbeater.cc:511] Master 127.27.196.126:39011 requested a full tablet report, sending...
I20240502 21:25:37.648434 28474 ts_manager.cc:194] Re-registered known tserver with Master: a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71:36827)
I20240502 21:25:37.649410 28474 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.71:60223
I20240502 21:25:37.958957 30353 heartbeater.cc:503] Master 127.27.196.126:39011 was elected leader, sending a full tablet report...
W20240502 21:25:38.085057 30490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:38.085587 30490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:38.085803 30490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:38.101523 30490 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:38.101748 30490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:38.101989 30490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.72
I20240502 21:25:38.120517 30490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.72:34997
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.27.196.72
--webserver_port=35791
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.27.196.126:39011
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.72
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:38.122709 30490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:38.125818 30490 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:38.145576 30495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:38.146780 30496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:38.152098 30490 server_base.cc:1034] running on GCE node
W20240502 21:25:38.153851 30499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:38.154857 30490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:38.156571 30490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:38.158053 30490 hybrid_clock.cc:648] HybridClock initialized: now 1714685138157960 us; error 81 us; skew 500 ppm
I20240502 21:25:38.158509 30490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:38.161495 30490 webserver.cc:460] Webserver started at http://127.27.196.72:35791/ using document root <none> and password file <none>
I20240502 21:25:38.162354 30490 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:38.162490 30490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:38.169384 30490 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240502 21:25:38.173051 30504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:38.174873 30490 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
I20240502 21:25:38.175099 30490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "4e8bd1e7c1f848b0bdd2570de97506d8"
format_stamp: "Formatted at 2024-05-02 21:25:24 on dist-test-slave-3bs1"
I20240502 21:25:38.176201 30490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:38.206035 30490 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:25:38.207437 30490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:38.207850 30490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:38.209470 30490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:38.211958 30490 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:38.212074 30490 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:38.212302 30490 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:38.212373 30490 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:25:38.262095 30490 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.72:34997
I20240502 21:25:38.262115 30617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.72:34997 every 8 connection(s)
I20240502 21:25:38.263967 30490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 21:25:38.265149 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 30490
I20240502 21:25:38.282835 30618 heartbeater.cc:347] Connected to a master server at 127.27.196.126:39011
I20240502 21:25:38.283345 30618 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:38.285315 30618 heartbeater.cc:511] Master 127.27.196.126:39011 requested a full tablet report, sending...
I20240502 21:25:38.289116 28476 ts_manager.cc:194] Re-registered known tserver with Master: 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72:34997)
I20240502 21:25:38.290536 28476 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.72:55907
I20240502 21:25:38.654018 30486 heartbeater.cc:503] Master 127.27.196.126:39011 was elected leader, sending a full tablet report...
I20240502 21:25:38.952903 28597 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:38.966370 30552 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:38.981288 28871 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:38.989177 30420 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:38.997210 28735 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:39.009824 29007 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:39.029287 30285 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:39.043097 30152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:39.295457 30618 heartbeater.cc:503] Master 127.27.196.126:39011 was elected leader, sending a full tablet report...
I20240502 21:25:39.769366 28475 ts_manager.cc:292] Set tserver state for 04e36797791842e489aee30de261c4c7 to MAINTENANCE_MODE
W20240502 21:25:41.378034 30692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:41.378500 30692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:41.389369 30692 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:25:41.607144 28597 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:41.671468 28871 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:41.672502 28597 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:41.682353 30552 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:41.707935 30552 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:41.709242 28735 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:41.744429 28871 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:41.753032 29007 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:41.753728 29006 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:41.768779 30152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:41.769613 28735 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:41.778615 30420 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:41.790135 30285 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:41.820073 30285 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:41.826442 30152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:41.848958 30420 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:42.138036 28597 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:42.154356 29006 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:42.174039 30552 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:42.175654 28871 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:42.223512 28735 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:42.261441 30152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:42.271813 30285 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:42.280055 30420 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:42.570233 28597 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:42.590662 29006 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:42.608868 28871 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:42.636548 28735 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:42.667533 30552 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:42.681460 30152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:42.722402 30420 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:42.724016 30285 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:42.829015 28893 consensus_queue.cc:237] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:42.832952 28755 raft_consensus.cc:1261] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:25:42.833161 28619 raft_consensus.cc:1261] T a4c6f9c73a2d4b01a2c027504e45b238 P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:25:42.833884 30763 consensus_queue.cc:1035] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:25:42.834501 30763 consensus_queue.cc:1035] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:25:42.838580 30763 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } }
I20240502 21:25:42.839447 28755 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } }
I20240502 21:25:42.847292 28471 catalog_manager.cc:5484] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } }
I20240502 21:25:42.862354 28619 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } }
I20240502 21:25:42.879346 28893 consensus_queue.cc:237] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } }
W20240502 21:25:42.882623 30819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:42.883277 30819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 21:25:42.885427 28755 raft_consensus.cc:1261] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:25:42.886027 28619 raft_consensus.cc:1261] T a4c6f9c73a2d4b01a2c027504e45b238 P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:25:42.891631 30868 consensus_queue.cc:1035] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:25:42.893558 30868 consensus_queue.cc:1035] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:25:42.900748 30868 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } } }
W20240502 21:25:42.902247 30819 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:25:42.911134 28467 catalog_manager.cc:5484] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 reported cstate change: config changed from index 4 to 5, NON_VOTER 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) added. New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:42.917553 28619 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } } }
I20240502 21:25:42.924407 28755 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } } }
I20240502 21:25:42.955626 28460 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a4c6f9c73a2d4b01a2c027504e45b238 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240502 21:25:42.959347 28823 consensus_peers.cc:485] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 -> Peer 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72:34997): Couldn't send request to peer 4e8bd1e7c1f848b0bdd2570de97506d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4c6f9c73a2d4b01a2c027504e45b238. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:43.139226 29006 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:43.144609 30552 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:43.156852 28871 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:43.161916 28597 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:43.164366 30285 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:43.178515 30152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:43.186024 28735 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:43.226995 30420 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:43.430209 30930 ts_tablet_manager.cc:902] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8: Initiating tablet copy from peer 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545)
I20240502 21:25:43.432899 30930 tablet_copy_client.cc:308] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:42545
I20240502 21:25:43.463843 28913 tablet_copy_service.cc:140] P 8733381e64da4e91a8a54e23b01e8967: Received BeginTabletCopySession request for tablet a4c6f9c73a2d4b01a2c027504e45b238 from peer 4e8bd1e7c1f848b0bdd2570de97506d8 ({username='slave'} at 127.27.196.72:35103)
I20240502 21:25:43.464135 28913 tablet_copy_service.cc:161] P 8733381e64da4e91a8a54e23b01e8967: Beginning new tablet copy session on tablet a4c6f9c73a2d4b01a2c027504e45b238 from peer 4e8bd1e7c1f848b0bdd2570de97506d8 at {username='slave'} at 127.27.196.72:35103: session id = 4e8bd1e7c1f848b0bdd2570de97506d8-a4c6f9c73a2d4b01a2c027504e45b238
I20240502 21:25:43.467864 28913 tablet_copy_source_session.cc:205] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:25:43.474506 30930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c6f9c73a2d4b01a2c027504e45b238. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:43.490511 30930 tablet_copy_client.cc:783] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8: tablet copy: Starting download of 0 data blocks...
I20240502 21:25:43.491003 30930 tablet_copy_client.cc:647] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8: tablet copy: Starting download of 1 WAL segments...
I20240502 21:25:43.493963 30934 tablet_copy_client.cc:1002] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8: tablet copy: Injecting latency into file download: 1500
W20240502 21:25:43.879523 30936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:43.879890 30936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:43.889530 30936 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:25:44.078387 30552 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:44.078385 28597 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:44.100764 28871 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:44.117424 29006 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:44.135306 30152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:44.178123 28735 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:44.178722 30285 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:44.180760 30420 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Runtime error: ksck discovered errors
I20240502 21:25:44.420015 30690 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 28536
I20240502 21:25:44.430189 28661 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:44.589848 30690 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 28536
W20240502 21:25:44.618404 28824 connection.cc:535] server connection from 127.27.196.65:59819 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:25:44.618477 28687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.196.65:44983 (error 108)
W20240502 21:25:44.618868 28824 connection.cc:535] client connection to 127.27.196.65:44983 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:25:44.619074 28688 connection.cc:535] server connection from 127.27.196.65:51509 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:25:44.619148 28824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:25:44.619237 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.196.65:44983 (error 108)
W20240502 21:25:44.623716 28687 consensus_peers.cc:485] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:44.623821 28824 consensus_peers.cc:485] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:44.624213 28961 consensus_peers.cc:485] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:44.673694 28687 consensus_peers.cc:485] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:44.687053 28824 consensus_peers.cc:485] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:44.745805 28687 consensus_peers.cc:485] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:44.847085 28961 consensus_peers.cc:485] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:44.867002 28824 consensus_peers.cc:485] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:44.971968 28824 consensus_peers.cc:485] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:44.994804 30930 tablet_copy_client.cc:523] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:25:44.999610 30930 tablet_bootstrap.cc:492] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8: Bootstrap starting.
I20240502 21:25:45.013935 30930 log.cc:824] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:45.050156 30930 tablet_bootstrap.cc:492] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:25:45.050709 30930 tablet_bootstrap.cc:492] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8: Bootstrap complete.
I20240502 21:25:45.051225 30930 ts_tablet_manager.cc:1367] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8: Time spent bootstrapping tablet: real 0.052s	user 0.051s	sys 0.000s
I20240502 21:25:45.056268 30930 raft_consensus.cc:348] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } }
I20240502 21:25:45.056721 30930 raft_consensus.cc:729] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e8bd1e7c1f848b0bdd2570de97506d8, State: Initialized, Role: LEARNER
I20240502 21:25:45.057284 30930 consensus_queue.cc:260] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } }
I20240502 21:25:45.059170 30930 ts_tablet_manager.cc:1398] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20240502 21:25:45.060735 28913 tablet_copy_service.cc:342] P 8733381e64da4e91a8a54e23b01e8967: Request end of tablet copy session 4e8bd1e7c1f848b0bdd2570de97506d8-a4c6f9c73a2d4b01a2c027504e45b238 received from {username='slave'} at 127.27.196.72:35103
I20240502 21:25:45.060954 28913 tablet_copy_service.cc:433] P 8733381e64da4e91a8a54e23b01e8967: ending tablet copy session 4e8bd1e7c1f848b0bdd2570de97506d8-a4c6f9c73a2d4b01a2c027504e45b238 on tablet a4c6f9c73a2d4b01a2c027504e45b238 with peer 4e8bd1e7c1f848b0bdd2570de97506d8
W20240502 21:25:45.064101 28824 consensus_peers.cc:485] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:45.393239 30572 raft_consensus.cc:1203] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 21:25:45.509145 31025 raft_consensus.cc:482] T d25783a912884d2ba72c586d6cd7dfdd P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 04e36797791842e489aee30de261c4c7)
I20240502 21:25:45.509384 31025 raft_consensus.cc:504] T d25783a912884d2ba72c586d6cd7dfdd P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:45.510242 31025 leader_election.cc:288] T d25783a912884d2ba72c586d6cd7dfdd P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795), 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983)
I20240502 21:25:45.511165 29029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25783a912884d2ba72c586d6cd7dfdd" candidate_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "52c5becf6262415cab0408b3968ed4c0" is_pre_election: true
W20240502 21:25:45.512594 28687 leader_election.cc:334] T d25783a912884d2ba72c586d6cd7dfdd P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
I20240502 21:25:45.512776 28687 leader_election.cc:302] T d25783a912884d2ba72c586d6cd7dfdd P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8786a1419d3f4bc79b88c54ce0675aa6; no voters: 04e36797791842e489aee30de261c4c7, 52c5becf6262415cab0408b3968ed4c0
I20240502 21:25:45.513073 31025 raft_consensus.cc:2726] T d25783a912884d2ba72c586d6cd7dfdd P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:25:45.702787 31036 raft_consensus.cc:482] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 04e36797791842e489aee30de261c4c7)
I20240502 21:25:45.703207 31036 raft_consensus.cc:504] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:45.704424 31036 leader_election.cc:288] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983), 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815)
I20240502 21:25:45.705804 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25783a912884d2ba72c586d6cd7dfdd" candidate_uuid: "52c5becf6262415cab0408b3968ed4c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" is_pre_election: true
I20240502 21:25:45.706351 28756 raft_consensus.cc:2454] T d25783a912884d2ba72c586d6cd7dfdd P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52c5becf6262415cab0408b3968ed4c0 in term 1.
W20240502 21:25:45.707213 28961 leader_election.cc:334] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
I20240502 21:25:45.707573 28961 leader_election.cc:302] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52c5becf6262415cab0408b3968ed4c0, 8786a1419d3f4bc79b88c54ce0675aa6; no voters: 04e36797791842e489aee30de261c4c7
I20240502 21:25:45.708001 31036 raft_consensus.cc:2781] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:25:45.708202 31036 raft_consensus.cc:482] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 04e36797791842e489aee30de261c4c7)
I20240502 21:25:45.708323 31036 raft_consensus.cc:3037] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:45.711473 31036 raft_consensus.cc:504] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:45.712424 31036 leader_election.cc:288] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 [CANDIDATE]: Term 2 election: Requested vote from peers 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983), 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815)
I20240502 21:25:45.713408 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25783a912884d2ba72c586d6cd7dfdd" candidate_uuid: "52c5becf6262415cab0408b3968ed4c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8786a1419d3f4bc79b88c54ce0675aa6"
I20240502 21:25:45.713677 28756 raft_consensus.cc:3037] T d25783a912884d2ba72c586d6cd7dfdd P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:25:45.713987 28961 leader_election.cc:334] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
I20240502 21:25:45.716574 28756 raft_consensus.cc:2454] T d25783a912884d2ba72c586d6cd7dfdd P 8786a1419d3f4bc79b88c54ce0675aa6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52c5becf6262415cab0408b3968ed4c0 in term 2.
I20240502 21:25:45.717396 28961 leader_election.cc:302] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52c5becf6262415cab0408b3968ed4c0, 8786a1419d3f4bc79b88c54ce0675aa6; no voters: 04e36797791842e489aee30de261c4c7
I20240502 21:25:45.717940 31036 raft_consensus.cc:2781] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:25:45.718156 31036 raft_consensus.cc:686] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 [term 2 LEADER]: Becoming Leader. State: Replica: 52c5becf6262415cab0408b3968ed4c0, State: Running, Role: LEADER
I20240502 21:25:45.718524 31036 consensus_queue.cc:237] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:45.722489 28471 catalog_manager.cc:5484] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 reported cstate change: term changed from 1 to 2, leader changed from 04e36797791842e489aee30de261c4c7 (127.27.196.65) to 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68). New cstate: current_term: 2 leader_uuid: "52c5becf6262415cab0408b3968ed4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:46.016966 31069 raft_consensus.cc:1050] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967: attempting to promote NON_VOTER 4e8bd1e7c1f848b0bdd2570de97506d8 to VOTER
I20240502 21:25:46.017978 31069 consensus_queue.cc:237] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } }
I20240502 21:25:46.021916 28756 raft_consensus.cc:1261] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:25:46.022063 30572 raft_consensus.cc:1261] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 LEARNER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 21:25:46.023047 28824 consensus_peers.cc:485] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:46.023075 31038 consensus_queue.cc:1035] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:25:46.023677 30965 consensus_queue.cc:1035] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:25:46.028780 31038 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:25:46.029865 30572 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:25:46.034386 28756 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:25:46.035712 28471 catalog_manager.cc:5484] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 reported cstate change: config changed from index 5 to 6, 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:46.041056 28892 consensus_queue.cc:237] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } }
I20240502 21:25:46.043206 28756 raft_consensus.cc:1261] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:25:46.043424 30572 raft_consensus.cc:1261] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:25:46.043920 31069 consensus_queue.cc:1035] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:25:46.044157 30872 consensus_queue.cc:1035] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:25:46.047801 30872 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 04e36797791842e489aee30de261c4c7 (127.27.196.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:25:46.048642 30572 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 04e36797791842e489aee30de261c4c7 (127.27.196.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:25:46.048671 28756 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 04e36797791842e489aee30de261c4c7 (127.27.196.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:25:46.052266 28460 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a4c6f9c73a2d4b01a2c027504e45b238 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:25:46.053200 28467 catalog_manager.cc:5484] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8 reported cstate change: config changed from index 6 to 7, VOTER 04e36797791842e489aee30de261c4c7 (127.27.196.65) evicted. New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
W20240502 21:25:46.059931 28461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
W20240502 21:25:46.062211 28461 catalog_manager.cc:4573] TS 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a4c6f9c73a2d4b01a2c027504e45b238: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
I20240502 21:25:46.091270 31025 raft_consensus.cc:482] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 04e36797791842e489aee30de261c4c7)
I20240502 21:25:46.091612 31025 raft_consensus.cc:504] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:46.092694 31025 leader_election.cc:288] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545), 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983)
I20240502 21:25:46.093685 31069 raft_consensus.cc:482] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 04e36797791842e489aee30de261c4c7)
I20240502 21:25:46.094089 28892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3665b7b9141b4fe8965eeecea35e9c0b" candidate_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8733381e64da4e91a8a54e23b01e8967" is_pre_election: true
I20240502 21:25:46.094471 31069 raft_consensus.cc:504] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:46.094897 28892 raft_consensus.cc:2454] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8786a1419d3f4bc79b88c54ce0675aa6 in term 1.
I20240502 21:25:46.095572 31069 leader_election.cc:288] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983), 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815)
W20240502 21:25:46.096668 28687 leader_election.cc:334] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
I20240502 21:25:46.097075 28686 leader_election.cc:302] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8733381e64da4e91a8a54e23b01e8967, 8786a1419d3f4bc79b88c54ce0675aa6; no voters: 04e36797791842e489aee30de261c4c7
W20240502 21:25:46.099165 28824 leader_election.cc:334] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
I20240502 21:25:46.099480 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3665b7b9141b4fe8965eeecea35e9c0b" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" is_pre_election: true
I20240502 21:25:46.097654 31025 raft_consensus.cc:2781] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:25:46.103533 28756 raft_consensus.cc:2454] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8733381e64da4e91a8a54e23b01e8967 in term 1.
I20240502 21:25:46.104779 28824 leader_election.cc:302] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8733381e64da4e91a8a54e23b01e8967, 8786a1419d3f4bc79b88c54ce0675aa6; no voters: 04e36797791842e489aee30de261c4c7
I20240502 21:25:46.105342 31069 raft_consensus.cc:2781] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:25:46.105348 31025 raft_consensus.cc:482] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 04e36797791842e489aee30de261c4c7)
I20240502 21:25:46.105525 31025 raft_consensus.cc:3037] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:46.105562 31069 raft_consensus.cc:482] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 04e36797791842e489aee30de261c4c7)
I20240502 21:25:46.105760 31069 raft_consensus.cc:3037] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:46.108435 31025 raft_consensus.cc:504] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:46.108695 31069 raft_consensus.cc:504] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:46.109459 31069 leader_election.cc:288] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 2 election: Requested vote from peers 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983), 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815)
I20240502 21:25:46.110124 28892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3665b7b9141b4fe8965eeecea35e9c0b" candidate_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8733381e64da4e91a8a54e23b01e8967"
I20240502 21:25:46.110440 28892 raft_consensus.cc:2379] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8786a1419d3f4bc79b88c54ce0675aa6 in current term 2: Already voted for candidate 8733381e64da4e91a8a54e23b01e8967 in this term.
I20240502 21:25:46.110865 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3665b7b9141b4fe8965eeecea35e9c0b" candidate_uuid: "8733381e64da4e91a8a54e23b01e8967" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8786a1419d3f4bc79b88c54ce0675aa6"
I20240502 21:25:46.111155 28756 raft_consensus.cc:2379] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8733381e64da4e91a8a54e23b01e8967 in current term 2: Already voted for candidate 8786a1419d3f4bc79b88c54ce0675aa6 in this term.
I20240502 21:25:46.111521 31025 leader_election.cc:288] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 2 election: Requested vote from peers 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545), 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983)
W20240502 21:25:46.112061 28824 leader_election.cc:334] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
I20240502 21:25:46.112191 28824 leader_election.cc:302] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8733381e64da4e91a8a54e23b01e8967; no voters: 04e36797791842e489aee30de261c4c7, 8786a1419d3f4bc79b88c54ce0675aa6
W20240502 21:25:46.112468 28687 leader_election.cc:334] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
I20240502 21:25:46.112586 28687 leader_election.cc:302] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8786a1419d3f4bc79b88c54ce0675aa6; no voters: 04e36797791842e489aee30de261c4c7, 8733381e64da4e91a8a54e23b01e8967
I20240502 21:25:46.112752 31069 raft_consensus.cc:2726] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 21:25:46.112958 31025 raft_consensus.cc:2726] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240502 21:25:46.153146 28961 consensus_peers.cc:485] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:46.196799 28756 raft_consensus.cc:1261] T d25783a912884d2ba72c586d6cd7dfdd P 8786a1419d3f4bc79b88c54ce0675aa6 [term 2 FOLLOWER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:25:46.197700 31036 consensus_queue.cc:1035] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:25:46.417268 28755 consensus_queue.cc:237] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } }
I20240502 21:25:46.421274 29029 raft_consensus.cc:1261] T a2fd105773d449da89940d3b6558361a P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Refusing update from remote peer 8786a1419d3f4bc79b88c54ce0675aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:25:46.422251 31091 consensus_queue.cc:1035] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:25:46.423012 28687 consensus_peers.cc:485] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:46.427017 31025 raft_consensus.cc:2932] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } }
I20240502 21:25:46.428006 29029 raft_consensus.cc:2932] T a2fd105773d449da89940d3b6558361a P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } }
I20240502 21:25:46.434077 28471 catalog_manager.cc:5484] T a2fd105773d449da89940d3b6558361a P 52c5becf6262415cab0408b3968ed4c0 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } }
I20240502 21:25:46.441949 28755 consensus_queue.cc:237] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: true } }
I20240502 21:25:46.446770 29029 raft_consensus.cc:1261] T a2fd105773d449da89940d3b6558361a P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Refusing update from remote peer 8786a1419d3f4bc79b88c54ce0675aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:25:46.447863 31098 consensus_queue.cc:1035] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:25:46.451035 31025 raft_consensus.cc:2932] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: true } } }
I20240502 21:25:46.457733 28461 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a2fd105773d449da89940d3b6558361a with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:25:46.458985 29029 raft_consensus.cc:2932] T a2fd105773d449da89940d3b6558361a P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: true } } }
W20240502 21:25:46.462337 28687 consensus_peers.cc:485] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69:41545): Couldn't send request to peer a1e2dba8e1a3403295cb394f7090f849. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a2fd105773d449da89940d3b6558361a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:46.464488 28467 catalog_manager.cc:5484] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 reported cstate change: config changed from index 4 to 5, NON_VOTER a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) added. New cstate: current_term: 1 leader_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:25:46.465090 28687 consensus_peers.cc:485] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:46.861599 28824 consensus_peers.cc:485] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:25:46.924031 28687 consensus_peers.cc:485] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:25:47.051072 31112 ts_tablet_manager.cc:902] T a2fd105773d449da89940d3b6558361a P a1e2dba8e1a3403295cb394f7090f849: Initiating tablet copy from peer 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815)
I20240502 21:25:47.053990 31112 tablet_copy_client.cc:308] T a2fd105773d449da89940d3b6558361a P a1e2dba8e1a3403295cb394f7090f849: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:41815
I20240502 21:25:47.064523 28777 tablet_copy_service.cc:140] P 8786a1419d3f4bc79b88c54ce0675aa6: Received BeginTabletCopySession request for tablet a2fd105773d449da89940d3b6558361a from peer a1e2dba8e1a3403295cb394f7090f849 ({username='slave'} at 127.27.196.69:33593)
I20240502 21:25:47.064921 28777 tablet_copy_service.cc:161] P 8786a1419d3f4bc79b88c54ce0675aa6: Beginning new tablet copy session on tablet a2fd105773d449da89940d3b6558361a from peer a1e2dba8e1a3403295cb394f7090f849 at {username='slave'} at 127.27.196.69:33593: session id = a1e2dba8e1a3403295cb394f7090f849-a2fd105773d449da89940d3b6558361a
I20240502 21:25:47.068642 28777 tablet_copy_source_session.cc:205] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:25:47.074581 31112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2fd105773d449da89940d3b6558361a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:47.088025 31112 tablet_copy_client.cc:783] T a2fd105773d449da89940d3b6558361a P a1e2dba8e1a3403295cb394f7090f849: tablet copy: Starting download of 0 data blocks...
I20240502 21:25:47.088454 31112 tablet_copy_client.cc:647] T a2fd105773d449da89940d3b6558361a P a1e2dba8e1a3403295cb394f7090f849: tablet copy: Starting download of 1 WAL segments...
I20240502 21:25:47.092341 31116 tablet_copy_client.cc:1002] T a2fd105773d449da89940d3b6558361a P a1e2dba8e1a3403295cb394f7090f849: tablet copy: Injecting latency into file download: 1500
W20240502 21:25:47.259069 28961 consensus_peers.cc:485] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:25:47.281730 28461 catalog_manager.cc:4573] TS 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a4c6f9c73a2d4b01a2c027504e45b238: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
W20240502 21:25:47.337519 28961 consensus_peers.cc:485] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:25:47.397276 28824 consensus_peers.cc:485] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:25:47.478417 28824 consensus_peers.cc:485] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:25:47.568161 28824 consensus_peers.cc:485] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:25:47.679788 28687 consensus_peers.cc:485] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:25:47.740262 31132 raft_consensus.cc:482] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:47.740554 31132 raft_consensus.cc:504] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:47.741750 31132 leader_election.cc:288] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545), 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983)
I20240502 21:25:47.742235 28892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3665b7b9141b4fe8965eeecea35e9c0b" candidate_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8733381e64da4e91a8a54e23b01e8967" is_pre_election: true
I20240502 21:25:47.742586 28892 raft_consensus.cc:2454] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8786a1419d3f4bc79b88c54ce0675aa6 in term 2.
W20240502 21:25:47.742790 28687 leader_election.cc:334] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
I20240502 21:25:47.743379 28686 leader_election.cc:302] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8733381e64da4e91a8a54e23b01e8967, 8786a1419d3f4bc79b88c54ce0675aa6; no voters: 04e36797791842e489aee30de261c4c7
I20240502 21:25:47.743853 31132 raft_consensus.cc:2781] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 21:25:47.744051 31132 raft_consensus.cc:482] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:47.744166 31132 raft_consensus.cc:3037] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:25:47.746975 31132 raft_consensus.cc:504] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:47.747807 31132 leader_election.cc:288] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 3 election: Requested vote from peers 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545), 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983)
I20240502 21:25:47.748628 28892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3665b7b9141b4fe8965eeecea35e9c0b" candidate_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8733381e64da4e91a8a54e23b01e8967"
I20240502 21:25:47.749004 28892 raft_consensus.cc:3037] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 2 FOLLOWER]: Advancing to term 3
W20240502 21:25:47.749975 28687 leader_election.cc:334] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
I20240502 21:25:47.751770 28892 raft_consensus.cc:2454] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8786a1419d3f4bc79b88c54ce0675aa6 in term 3.
I20240502 21:25:47.752192 28686 leader_election.cc:302] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8733381e64da4e91a8a54e23b01e8967, 8786a1419d3f4bc79b88c54ce0675aa6; no voters: 04e36797791842e489aee30de261c4c7
I20240502 21:25:47.752545 31132 raft_consensus.cc:2781] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:25:47.753134 31132 raft_consensus.cc:686] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 3 LEADER]: Becoming Leader. State: Replica: 8786a1419d3f4bc79b88c54ce0675aa6, State: Running, Role: LEADER
I20240502 21:25:47.753451 31132 consensus_queue.cc:237] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:47.757174 28467 catalog_manager.cc:5484] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 reported cstate change: term changed from 1 to 3, leader changed from 04e36797791842e489aee30de261c4c7 (127.27.196.65) to 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66). New cstate: current_term: 3 leader_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:48.289505 28892 raft_consensus.cc:1261] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 3 FOLLOWER]: Refusing update from remote peer 8786a1419d3f4bc79b88c54ce0675aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 21:25:48.290184 31131 consensus_queue.cc:1035] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:25:48.297524 28687 consensus_peers.cc:485] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:48.351683 28461 catalog_manager.cc:4573] TS 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a4c6f9c73a2d4b01a2c027504e45b238: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
I20240502 21:25:48.593502 31112 tablet_copy_client.cc:523] T a2fd105773d449da89940d3b6558361a P a1e2dba8e1a3403295cb394f7090f849: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:25:48.598098 31112 tablet_bootstrap.cc:492] T a2fd105773d449da89940d3b6558361a P a1e2dba8e1a3403295cb394f7090f849: Bootstrap starting.
I20240502 21:25:48.610466 31112 log.cc:824] T a2fd105773d449da89940d3b6558361a P a1e2dba8e1a3403295cb394f7090f849: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:48.645453 31112 tablet_bootstrap.cc:492] T a2fd105773d449da89940d3b6558361a P a1e2dba8e1a3403295cb394f7090f849: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:25:48.646111 31112 tablet_bootstrap.cc:492] T a2fd105773d449da89940d3b6558361a P a1e2dba8e1a3403295cb394f7090f849: Bootstrap complete.
I20240502 21:25:48.646615 31112 ts_tablet_manager.cc:1367] T a2fd105773d449da89940d3b6558361a P a1e2dba8e1a3403295cb394f7090f849: Time spent bootstrapping tablet: real 0.049s	user 0.047s	sys 0.001s
W20240502 21:25:48.650487 28961 consensus_peers.cc:485] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:25:48.651942 31112 raft_consensus.cc:348] T a2fd105773d449da89940d3b6558361a P a1e2dba8e1a3403295cb394f7090f849 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: true } }
I20240502 21:25:48.652395 31112 raft_consensus.cc:729] T a2fd105773d449da89940d3b6558361a P a1e2dba8e1a3403295cb394f7090f849 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a1e2dba8e1a3403295cb394f7090f849, State: Initialized, Role: LEARNER
I20240502 21:25:48.652945 31112 consensus_queue.cc:260] T a2fd105773d449da89940d3b6558361a P a1e2dba8e1a3403295cb394f7090f849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: true } }
I20240502 21:25:48.654775 31112 ts_tablet_manager.cc:1398] T a2fd105773d449da89940d3b6558361a P a1e2dba8e1a3403295cb394f7090f849: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.004s
I20240502 21:25:48.656344 28777 tablet_copy_service.cc:342] P 8786a1419d3f4bc79b88c54ce0675aa6: Request end of tablet copy session a1e2dba8e1a3403295cb394f7090f849-a2fd105773d449da89940d3b6558361a received from {username='slave'} at 127.27.196.69:33593
I20240502 21:25:48.656535 28777 tablet_copy_service.cc:433] P 8786a1419d3f4bc79b88c54ce0675aa6: ending tablet copy session a1e2dba8e1a3403295cb394f7090f849-a2fd105773d449da89940d3b6558361a on tablet a2fd105773d449da89940d3b6558361a with peer a1e2dba8e1a3403295cb394f7090f849
W20240502 21:25:48.740737 28687 consensus_peers.cc:485] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:25:49.069918 30172 raft_consensus.cc:1203] T a2fd105773d449da89940d3b6558361a P a1e2dba8e1a3403295cb394f7090f849 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
W20240502 21:25:49.285647 28687 consensus_peers.cc:485] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:25:49.335211 28824 consensus_peers.cc:485] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:25:49.603700 28961 consensus_peers.cc:485] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:25:49.678565 31205 raft_consensus.cc:1050] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6: attempting to promote NON_VOTER a1e2dba8e1a3403295cb394f7090f849 to VOTER
I20240502 21:25:49.679713 31205 consensus_queue.cc:237] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } }
I20240502 21:25:49.682873 29029 raft_consensus.cc:1261] T a2fd105773d449da89940d3b6558361a P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Refusing update from remote peer 8786a1419d3f4bc79b88c54ce0675aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:25:49.683156 30172 raft_consensus.cc:1261] T a2fd105773d449da89940d3b6558361a P a1e2dba8e1a3403295cb394f7090f849 [term 1 LEARNER]: Refusing update from remote peer 8786a1419d3f4bc79b88c54ce0675aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:25:49.683655 31207 consensus_queue.cc:1035] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 21:25:49.683794 28687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 21:25:49.684340 31207 consensus_queue.cc:1035] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 21:25:49.685266 28687 consensus_peers.cc:485] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:49.689530 31205 raft_consensus.cc:2932] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
I20240502 21:25:49.690515 30172 raft_consensus.cc:2932] T a2fd105773d449da89940d3b6558361a P a1e2dba8e1a3403295cb394f7090f849 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
I20240502 21:25:49.694331 29029 raft_consensus.cc:2932] T a2fd105773d449da89940d3b6558361a P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
I20240502 21:25:49.695432 28471 catalog_manager.cc:5484] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 reported cstate change: config changed from index 5 to 6, a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:49.703817 28755 consensus_queue.cc:237] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } }
I20240502 21:25:49.706288 30172 raft_consensus.cc:1261] T a2fd105773d449da89940d3b6558361a P a1e2dba8e1a3403295cb394f7090f849 [term 1 FOLLOWER]: Refusing update from remote peer 8786a1419d3f4bc79b88c54ce0675aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:25:49.706288 29029 raft_consensus.cc:1261] T a2fd105773d449da89940d3b6558361a P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Refusing update from remote peer 8786a1419d3f4bc79b88c54ce0675aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:25:49.707228 31160 consensus_queue.cc:1035] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:25:49.707935 31160 consensus_queue.cc:1035] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:25:49.709944 31205 raft_consensus.cc:2932] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 04e36797791842e489aee30de261c4c7 (127.27.196.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
W20240502 21:25:49.710742 28824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 46 similar messages]
I20240502 21:25:49.711485 30172 raft_consensus.cc:2932] T a2fd105773d449da89940d3b6558361a P a1e2dba8e1a3403295cb394f7090f849 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 04e36797791842e489aee30de261c4c7 (127.27.196.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
I20240502 21:25:49.714346 28461 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a2fd105773d449da89940d3b6558361a with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240502 21:25:49.715080 28824 consensus_peers.cc:485] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:25:49.717033 28475 catalog_manager.cc:5484] T a2fd105773d449da89940d3b6558361a P 8786a1419d3f4bc79b88c54ce0675aa6 reported cstate change: config changed from index 6 to 7, VOTER 04e36797791842e489aee30de261c4c7 (127.27.196.65) evicted. New cstate: current_term: 1 leader_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:49.720865 29029 raft_consensus.cc:2932] T a2fd105773d449da89940d3b6558361a P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 04e36797791842e489aee30de261c4c7 (127.27.196.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
W20240502 21:25:49.724086 28461 catalog_manager.cc:4573] TS 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a2fd105773d449da89940d3b6558361a: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
W20240502 21:25:49.737342 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 21:25:49.738950 28961 consensus_peers.cc:485] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:25:49.808207 28824 consensus_peers.cc:485] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:25:49.969918 28824 consensus_peers.cc:485] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:25:50.109208 28892 consensus_queue.cc:237] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:25:50.112140 29029 raft_consensus.cc:1261] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:25:50.113000 31176 consensus_queue.cc:1035] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:25:50.113353 28824 consensus_peers.cc:485] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:50.116605 31198 raft_consensus.cc:2932] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } }
I20240502 21:25:50.117429 29029 raft_consensus.cc:2932] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } }
I20240502 21:25:50.122371 28475 catalog_manager.cc:5484] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } }
I20240502 21:25:50.130865 28892 consensus_queue.cc:237] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } }
I20240502 21:25:50.134574 29029 raft_consensus.cc:1261] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:25:50.135744 31176 consensus_queue.cc:1035] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:25:50.140504 31198 raft_consensus.cc:2932] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } } }
I20240502 21:25:50.141620 29029 raft_consensus.cc:2932] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } } }
W20240502 21:25:50.148098 28826 consensus_peers.cc:485] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 -> Peer fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70:43969): Couldn't send request to peer fc5aa2dc1f7b4554906785d693e272d0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a58681ae3e54019aefa430aaa3dc046. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:50.149317 28460 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4a58681ae3e54019aefa430aaa3dc046 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:25:50.150126 28467 catalog_manager.cc:5484] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0 reported cstate change: config changed from index 4 to 5, NON_VOTER fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) added. New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } } }
W20240502 21:25:50.151206 28824 consensus_peers.cc:485] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:50.194670 28687 consensus_peers.cc:485] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:25:50.649187 31250 ts_tablet_manager.cc:902] T 4a58681ae3e54019aefa430aaa3dc046 P fc5aa2dc1f7b4554906785d693e272d0: Initiating tablet copy from peer 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545)
I20240502 21:25:50.651288 31250 tablet_copy_client.cc:308] T 4a58681ae3e54019aefa430aaa3dc046 P fc5aa2dc1f7b4554906785d693e272d0: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:42545
I20240502 21:25:50.659071 28913 tablet_copy_service.cc:140] P 8733381e64da4e91a8a54e23b01e8967: Received BeginTabletCopySession request for tablet 4a58681ae3e54019aefa430aaa3dc046 from peer fc5aa2dc1f7b4554906785d693e272d0 ({username='slave'} at 127.27.196.70:40429)
I20240502 21:25:50.659379 28913 tablet_copy_service.cc:161] P 8733381e64da4e91a8a54e23b01e8967: Beginning new tablet copy session on tablet 4a58681ae3e54019aefa430aaa3dc046 from peer fc5aa2dc1f7b4554906785d693e272d0 at {username='slave'} at 127.27.196.70:40429: session id = fc5aa2dc1f7b4554906785d693e272d0-4a58681ae3e54019aefa430aaa3dc046
I20240502 21:25:50.661939 28913 tablet_copy_source_session.cc:205] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:25:50.666823 31250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a58681ae3e54019aefa430aaa3dc046. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:50.678226 31250 tablet_copy_client.cc:783] T 4a58681ae3e54019aefa430aaa3dc046 P fc5aa2dc1f7b4554906785d693e272d0: tablet copy: Starting download of 0 data blocks...
I20240502 21:25:50.678651 31250 tablet_copy_client.cc:647] T 4a58681ae3e54019aefa430aaa3dc046 P fc5aa2dc1f7b4554906785d693e272d0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:25:50.681702 31254 tablet_copy_client.cc:1002] T 4a58681ae3e54019aefa430aaa3dc046 P fc5aa2dc1f7b4554906785d693e272d0: tablet copy: Injecting latency into file download: 1500
W20240502 21:25:50.872645 28461 catalog_manager.cc:4573] TS 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a2fd105773d449da89940d3b6558361a: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
W20240502 21:25:51.014734 28687 consensus_peers.cc:485] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:25:51.298070 28961 consensus_peers.cc:485] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:25:51.901177 28687 consensus_peers.cc:485] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:25:51.939685 28461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 15 similar messages]
W20240502 21:25:51.941838 28461 catalog_manager.cc:4573] TS 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a2fd105773d449da89940d3b6558361a: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
W20240502 21:25:52.050577 28961 consensus_peers.cc:485] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:25:52.131037 28824 consensus_peers.cc:485] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 21:25:52.182689 31250 tablet_copy_client.cc:523] T 4a58681ae3e54019aefa430aaa3dc046 P fc5aa2dc1f7b4554906785d693e272d0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:25:52.188441 31250 tablet_bootstrap.cc:492] T 4a58681ae3e54019aefa430aaa3dc046 P fc5aa2dc1f7b4554906785d693e272d0: Bootstrap starting.
I20240502 21:25:52.201594 31250 log.cc:824] T 4a58681ae3e54019aefa430aaa3dc046 P fc5aa2dc1f7b4554906785d693e272d0: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:52.238312 31250 tablet_bootstrap.cc:492] T 4a58681ae3e54019aefa430aaa3dc046 P fc5aa2dc1f7b4554906785d693e272d0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:25:52.238880 31250 tablet_bootstrap.cc:492] T 4a58681ae3e54019aefa430aaa3dc046 P fc5aa2dc1f7b4554906785d693e272d0: Bootstrap complete.
I20240502 21:25:52.239461 31250 ts_tablet_manager.cc:1367] T 4a58681ae3e54019aefa430aaa3dc046 P fc5aa2dc1f7b4554906785d693e272d0: Time spent bootstrapping tablet: real 0.051s	user 0.048s	sys 0.004s
I20240502 21:25:52.245065 31250 raft_consensus.cc:348] T 4a58681ae3e54019aefa430aaa3dc046 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } }
I20240502 21:25:52.245513 31250 raft_consensus.cc:729] T 4a58681ae3e54019aefa430aaa3dc046 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fc5aa2dc1f7b4554906785d693e272d0, State: Initialized, Role: LEARNER
I20240502 21:25:52.246081 31250 consensus_queue.cc:260] T 4a58681ae3e54019aefa430aaa3dc046 P fc5aa2dc1f7b4554906785d693e272d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } }
I20240502 21:25:52.247736 31250 ts_tablet_manager.cc:1398] T 4a58681ae3e54019aefa430aaa3dc046 P fc5aa2dc1f7b4554906785d693e272d0: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20240502 21:25:52.248997 28913 tablet_copy_service.cc:342] P 8733381e64da4e91a8a54e23b01e8967: Request end of tablet copy session fc5aa2dc1f7b4554906785d693e272d0-4a58681ae3e54019aefa430aaa3dc046 received from {username='slave'} at 127.27.196.70:40429
I20240502 21:25:52.249150 28913 tablet_copy_service.cc:433] P 8733381e64da4e91a8a54e23b01e8967: ending tablet copy session fc5aa2dc1f7b4554906785d693e272d0-4a58681ae3e54019aefa430aaa3dc046 on tablet 4a58681ae3e54019aefa430aaa3dc046 with peer fc5aa2dc1f7b4554906785d693e272d0
W20240502 21:25:52.318701 28961 consensus_peers.cc:485] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:25:52.491556 28824 consensus_peers.cc:485] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:25:52.577224 28687 consensus_peers.cc:485] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:25:52.614248 28824 consensus_peers.cc:485] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 21:25:52.674239 30305 raft_consensus.cc:1203] T 4a58681ae3e54019aefa430aaa3dc046 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
W20240502 21:25:52.712193 28824 consensus_peers.cc:485] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:25:53.128966 31286 raft_consensus.cc:1050] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967: attempting to promote NON_VOTER fc5aa2dc1f7b4554906785d693e272d0 to VOTER
I20240502 21:25:53.129784 31286 consensus_queue.cc:237] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } }
I20240502 21:25:53.133241 29029 raft_consensus.cc:1261] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:25:53.133255 30305 raft_consensus.cc:1261] T 4a58681ae3e54019aefa430aaa3dc046 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 LEARNER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 21:25:53.133574 28824 consensus_peers.cc:485] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:53.133908 31286 consensus_queue.cc:1035] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:25:53.134351 31325 consensus_queue.cc:1035] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:25:53.141274 31286 raft_consensus.cc:2932] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:25:53.142904 30305 raft_consensus.cc:2932] T 4a58681ae3e54019aefa430aaa3dc046 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:25:53.148996 28475 catalog_manager.cc:5484] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 reported cstate change: config changed from index 5 to 6, fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:53.149224 29029 raft_consensus.cc:2932] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:25:53.158610 28893 consensus_queue.cc:237] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } }
I20240502 21:25:53.161430 30305 raft_consensus.cc:1261] T 4a58681ae3e54019aefa430aaa3dc046 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:25:53.162343 31325 consensus_queue.cc:1035] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:25:53.162986 29029 raft_consensus.cc:1261] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:25:53.164402 31325 consensus_queue.cc:1035] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:25:53.165856 31322 raft_consensus.cc:2932] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 04e36797791842e489aee30de261c4c7 (127.27.196.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:25:53.166515 30305 raft_consensus.cc:2932] T 4a58681ae3e54019aefa430aaa3dc046 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 04e36797791842e489aee30de261c4c7 (127.27.196.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:25:53.169487 28460 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4a58681ae3e54019aefa430aaa3dc046 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:25:53.171419 28467 catalog_manager.cc:5484] T 4a58681ae3e54019aefa430aaa3dc046 P fc5aa2dc1f7b4554906785d693e272d0 reported cstate change: config changed from index 6 to 7, VOTER 04e36797791842e489aee30de261c4c7 (127.27.196.65) evicted. New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:25:53.171617 29029 raft_consensus.cc:2932] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 04e36797791842e489aee30de261c4c7 (127.27.196.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
W20240502 21:25:53.177632 28461 catalog_manager.cc:4573] TS 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4a58681ae3e54019aefa430aaa3dc046: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
W20240502 21:25:53.361259 28687 consensus_peers.cc:485] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:25:53.592123 28755 consensus_queue.cc:237] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:53.595937 28893 raft_consensus.cc:1261] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 3 FOLLOWER]: Refusing update from remote peer 8786a1419d3f4bc79b88c54ce0675aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20240502 21:25:53.596532 28687 consensus_peers.cc:485] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:53.596771 31351 consensus_queue.cc:1035] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:25:53.601083 31249 raft_consensus.cc:2932] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } }
I20240502 21:25:53.601966 28893 raft_consensus.cc:2932] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } }
I20240502 21:25:53.609470 28467 catalog_manager.cc:5484] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 reported cstate change: config changed from index -1 to 5. New cstate: current_term: 3 leader_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } }
I20240502 21:25:53.614995 28755 consensus_queue.cc:237] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } }
I20240502 21:25:53.618450 28893 raft_consensus.cc:1261] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 3 FOLLOWER]: Refusing update from remote peer 8786a1419d3f4bc79b88c54ce0675aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240502 21:25:53.619256 31359 consensus_queue.cc:1035] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 21:25:53.623859 28687 consensus_peers.cc:485] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:53.623703 31249 raft_consensus.cc:2932] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, NON_VOTER fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } } }
W20240502 21:25:53.626684 28689 consensus_peers.cc:485] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70:43969): Couldn't send request to peer fc5aa2dc1f7b4554906785d693e272d0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3665b7b9141b4fe8965eeecea35e9c0b. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:53.628206 28461 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3665b7b9141b4fe8965eeecea35e9c0b with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:25:53.628878 28471 catalog_manager.cc:5484] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 reported cstate change: config changed from index 5 to 6, NON_VOTER fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) added. New cstate: current_term: 3 leader_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:53.631563 28893 raft_consensus.cc:2932] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, NON_VOTER fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } } }
W20240502 21:25:53.683913 28961 consensus_peers.cc:485] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 21:25:54.096130 31372 ts_tablet_manager.cc:902] T 3665b7b9141b4fe8965eeecea35e9c0b P fc5aa2dc1f7b4554906785d693e272d0: Initiating tablet copy from peer 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815)
I20240502 21:25:54.097636 31372 tablet_copy_client.cc:308] T 3665b7b9141b4fe8965eeecea35e9c0b P fc5aa2dc1f7b4554906785d693e272d0: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:41815
I20240502 21:25:54.105005 28777 tablet_copy_service.cc:140] P 8786a1419d3f4bc79b88c54ce0675aa6: Received BeginTabletCopySession request for tablet 3665b7b9141b4fe8965eeecea35e9c0b from peer fc5aa2dc1f7b4554906785d693e272d0 ({username='slave'} at 127.27.196.70:44087)
I20240502 21:25:54.105252 28777 tablet_copy_service.cc:161] P 8786a1419d3f4bc79b88c54ce0675aa6: Beginning new tablet copy session on tablet 3665b7b9141b4fe8965eeecea35e9c0b from peer fc5aa2dc1f7b4554906785d693e272d0 at {username='slave'} at 127.27.196.70:44087: session id = fc5aa2dc1f7b4554906785d693e272d0-3665b7b9141b4fe8965eeecea35e9c0b
I20240502 21:25:54.107573 28777 tablet_copy_source_session.cc:205] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:25:54.109277 31372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3665b7b9141b4fe8965eeecea35e9c0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:54.114236 31372 tablet_copy_client.cc:783] T 3665b7b9141b4fe8965eeecea35e9c0b P fc5aa2dc1f7b4554906785d693e272d0: tablet copy: Starting download of 0 data blocks...
I20240502 21:25:54.114539 31372 tablet_copy_client.cc:647] T 3665b7b9141b4fe8965eeecea35e9c0b P fc5aa2dc1f7b4554906785d693e272d0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:25:54.117192 31376 tablet_copy_client.cc:1002] T 3665b7b9141b4fe8965eeecea35e9c0b P fc5aa2dc1f7b4554906785d693e272d0: tablet copy: Injecting latency into file download: 1500
W20240502 21:25:54.324975 28961 consensus_peers.cc:485] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:25:54.328574 28687 consensus_peers.cc:485] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:25:54.336241 28461 catalog_manager.cc:4573] TS 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4a58681ae3e54019aefa430aaa3dc046: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
W20240502 21:25:54.646297 28824 consensus_peers.cc:485] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:25:54.772715 28824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 21:25:54.773525 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 21:25:54.773785 28824 consensus_peers.cc:485] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:25:54.830163 28687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240502 21:25:54.999083 28687 consensus_peers.cc:485] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:25:55.089278 28824 consensus_peers.cc:485] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:25:55.145888 28961 consensus_peers.cc:485] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:25:55.393841 28461 catalog_manager.cc:4573] TS 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4a58681ae3e54019aefa430aaa3dc046: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
I20240502 21:25:55.617911 31372 tablet_copy_client.cc:523] T 3665b7b9141b4fe8965eeecea35e9c0b P fc5aa2dc1f7b4554906785d693e272d0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:25:55.621244 31372 tablet_bootstrap.cc:492] T 3665b7b9141b4fe8965eeecea35e9c0b P fc5aa2dc1f7b4554906785d693e272d0: Bootstrap starting.
I20240502 21:25:55.657335 31372 tablet_bootstrap.cc:492] T 3665b7b9141b4fe8965eeecea35e9c0b P fc5aa2dc1f7b4554906785d693e272d0: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:25:55.657789 31372 tablet_bootstrap.cc:492] T 3665b7b9141b4fe8965eeecea35e9c0b P fc5aa2dc1f7b4554906785d693e272d0: Bootstrap complete.
I20240502 21:25:55.658075 31372 ts_tablet_manager.cc:1367] T 3665b7b9141b4fe8965eeecea35e9c0b P fc5aa2dc1f7b4554906785d693e272d0: Time spent bootstrapping tablet: real 0.037s	user 0.032s	sys 0.003s
I20240502 21:25:55.659003 31372 raft_consensus.cc:348] T 3665b7b9141b4fe8965eeecea35e9c0b P fc5aa2dc1f7b4554906785d693e272d0 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } }
I20240502 21:25:55.659288 31372 raft_consensus.cc:729] T 3665b7b9141b4fe8965eeecea35e9c0b P fc5aa2dc1f7b4554906785d693e272d0 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: fc5aa2dc1f7b4554906785d693e272d0, State: Initialized, Role: LEARNER
I20240502 21:25:55.659507 31372 consensus_queue.cc:260] T 3665b7b9141b4fe8965eeecea35e9c0b P fc5aa2dc1f7b4554906785d693e272d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } }
I20240502 21:25:55.660432 31372 ts_tablet_manager.cc:1398] T 3665b7b9141b4fe8965eeecea35e9c0b P fc5aa2dc1f7b4554906785d693e272d0: Time spent starting tablet: real 0.002s	user 0.005s	sys 0.000s
I20240502 21:25:55.661387 28777 tablet_copy_service.cc:342] P 8786a1419d3f4bc79b88c54ce0675aa6: Request end of tablet copy session fc5aa2dc1f7b4554906785d693e272d0-3665b7b9141b4fe8965eeecea35e9c0b received from {username='slave'} at 127.27.196.70:44087
I20240502 21:25:55.661573 28777 tablet_copy_service.cc:433] P 8786a1419d3f4bc79b88c54ce0675aa6: ending tablet copy session fc5aa2dc1f7b4554906785d693e272d0-3665b7b9141b4fe8965eeecea35e9c0b on tablet 3665b7b9141b4fe8965eeecea35e9c0b with peer fc5aa2dc1f7b4554906785d693e272d0
I20240502 21:25:55.988878 30305 raft_consensus.cc:1203] T 3665b7b9141b4fe8965eeecea35e9c0b P fc5aa2dc1f7b4554906785d693e272d0 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.5->[3.6-3.6]   Dedup: 3.6->[]
W20240502 21:25:56.150274 28687 consensus_peers.cc:485] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:25:56.333177 28961 consensus_peers.cc:485] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 21:25:56.539664 31441 raft_consensus.cc:1050] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6: attempting to promote NON_VOTER fc5aa2dc1f7b4554906785d693e272d0 to VOTER
I20240502 21:25:56.540414 31441 consensus_queue.cc:237] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } }
I20240502 21:25:56.543395 30305 raft_consensus.cc:1261] T 3665b7b9141b4fe8965eeecea35e9c0b P fc5aa2dc1f7b4554906785d693e272d0 [term 3 LEARNER]: Refusing update from remote peer 8786a1419d3f4bc79b88c54ce0675aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240502 21:25:56.543977 28893 raft_consensus.cc:1261] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 3 FOLLOWER]: Refusing update from remote peer 8786a1419d3f4bc79b88c54ce0675aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240502 21:25:56.544178 31441 consensus_queue.cc:1035] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 21:25:56.544570 28687 consensus_peers.cc:485] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:56.544652 31444 consensus_queue.cc:1035] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:25:56.547884 31441 raft_consensus.cc:2932] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 3 LEADER]: Committing config change with OpId 3.7: config changed from index 6 to 7, fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:25:56.549015 28893 raft_consensus.cc:2932] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 6 to 7, fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:25:56.551993 30305 raft_consensus.cc:2932] T 3665b7b9141b4fe8965eeecea35e9c0b P fc5aa2dc1f7b4554906785d693e272d0 [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 6 to 7, fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:25:56.553287 28471 catalog_manager.cc:5484] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 reported cstate change: config changed from index 6 to 7, fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:56.564929 28755 consensus_queue.cc:237] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } }
I20240502 21:25:56.567260 28893 raft_consensus.cc:1261] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 3 FOLLOWER]: Refusing update from remote peer 8786a1419d3f4bc79b88c54ce0675aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240502 21:25:56.567490 30305 raft_consensus.cc:1261] T 3665b7b9141b4fe8965eeecea35e9c0b P fc5aa2dc1f7b4554906785d693e272d0 [term 3 FOLLOWER]: Refusing update from remote peer 8786a1419d3f4bc79b88c54ce0675aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240502 21:25:56.567991 31359 consensus_queue.cc:1035] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:25:56.568295 31441 consensus_queue.cc:1035] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:25:56.571053 31442 raft_consensus.cc:2932] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 3 LEADER]: Committing config change with OpId 3.8: config changed from index 7 to 8, VOTER 04e36797791842e489aee30de261c4c7 (127.27.196.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:25:56.571828 28893 raft_consensus.cc:2932] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 7 to 8, VOTER 04e36797791842e489aee30de261c4c7 (127.27.196.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:25:56.572005 30305 raft_consensus.cc:2932] T 3665b7b9141b4fe8965eeecea35e9c0b P fc5aa2dc1f7b4554906785d693e272d0 [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 7 to 8, VOTER 04e36797791842e489aee30de261c4c7 (127.27.196.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:25:56.575870 28461 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3665b7b9141b4fe8965eeecea35e9c0b with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:25:56.576294 28467 catalog_manager.cc:5484] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 reported cstate change: config changed from index 7 to 8, VOTER 04e36797791842e489aee30de261c4c7 (127.27.196.65) evicted. New cstate: current_term: 3 leader_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 21:25:56.584414 28461 catalog_manager.cc:4573] TS 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3665b7b9141b4fe8965eeecea35e9c0b: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
I20240502 21:25:56.831662 29029 consensus_queue.cc:237] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:25:56.834527 28755 raft_consensus.cc:1261] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:25:56.835479 28961 consensus_peers.cc:485] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:56.835531 31466 consensus_queue.cc:1035] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:25:56.839819 31197 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } }
I20240502 21:25:56.840691 28755 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } }
I20240502 21:25:56.847913 28471 catalog_manager.cc:5484] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "52c5becf6262415cab0408b3968ed4c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } }
I20240502 21:25:56.855538 29029 consensus_queue.cc:237] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: NON_VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: true } }
I20240502 21:25:56.860579 28756 raft_consensus.cc:1261] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:25:56.863729 31468 consensus_queue.cc:1035] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 21:25:56.864171 28961 consensus_peers.cc:485] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:56.866873 31197 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: NON_VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: true } } }
I20240502 21:25:56.871011 28462 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 876004fec43e428e930f3fbb873ccac8 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:25:56.871778 28756 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: NON_VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: true } } }
I20240502 21:25:56.873602 28475 catalog_manager.cc:5484] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 reported cstate change: config changed from index 4 to 5, NON_VOTER a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) added. New cstate: current_term: 1 leader_uuid: "52c5becf6262415cab0408b3968ed4c0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: NON_VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:25:56.884353 28962 consensus_peers.cc:485] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 -> Peer a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71:36827): Couldn't send request to peer a9ca254f89984b3f9ed62020bd6a4d02. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 876004fec43e428e930f3fbb873ccac8. This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:56.910187 28687 consensus_peers.cc:485] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:25:57.116775 28824 consensus_peers.cc:485] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:25:57.200171 28461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 15 similar messages]
W20240502 21:25:57.331897 28824 consensus_peers.cc:485] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 21:25:57.359773 31487 ts_tablet_manager.cc:902] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02: Initiating tablet copy from peer 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795)
I20240502 21:25:57.361907 31487 tablet_copy_client.cc:308] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.68:41795
I20240502 21:25:57.370630 29050 tablet_copy_service.cc:140] P 52c5becf6262415cab0408b3968ed4c0: Received BeginTabletCopySession request for tablet 876004fec43e428e930f3fbb873ccac8 from peer a9ca254f89984b3f9ed62020bd6a4d02 ({username='slave'} at 127.27.196.71:34711)
I20240502 21:25:57.370947 29050 tablet_copy_service.cc:161] P 52c5becf6262415cab0408b3968ed4c0: Beginning new tablet copy session on tablet 876004fec43e428e930f3fbb873ccac8 from peer a9ca254f89984b3f9ed62020bd6a4d02 at {username='slave'} at 127.27.196.71:34711: session id = a9ca254f89984b3f9ed62020bd6a4d02-876004fec43e428e930f3fbb873ccac8
I20240502 21:25:57.373925 29050 tablet_copy_source_session.cc:205] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:25:57.379609 31487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 876004fec43e428e930f3fbb873ccac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:57.395505 31487 tablet_copy_client.cc:783] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02: tablet copy: Starting download of 0 data blocks...
I20240502 21:25:57.395901 31487 tablet_copy_client.cc:647] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02: tablet copy: Starting download of 1 WAL segments...
I20240502 21:25:57.398244 31491 tablet_copy_client.cc:1002] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02: tablet copy: Injecting latency into file download: 1500
W20240502 21:25:57.459620 28687 consensus_peers.cc:485] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:25:57.594636 28824 consensus_peers.cc:485] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:25:57.612210 28961 consensus_peers.cc:485] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:25:57.724593 28461 catalog_manager.cc:4573] TS 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3665b7b9141b4fe8965eeecea35e9c0b: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
W20240502 21:25:58.769944 28461 catalog_manager.cc:4573] TS 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3665b7b9141b4fe8965eeecea35e9c0b: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
W20240502 21:25:58.875978 28961 consensus_peers.cc:485] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 21:25:58.899181 31487 tablet_copy_client.cc:523] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:25:58.903546 31487 tablet_bootstrap.cc:492] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02: Bootstrap starting.
I20240502 21:25:58.914809 31487 log.cc:824] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:58.952278 31487 tablet_bootstrap.cc:492] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:25:58.953313 31487 tablet_bootstrap.cc:492] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02: Bootstrap complete.
I20240502 21:25:58.954162 31487 ts_tablet_manager.cc:1367] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02: Time spent bootstrapping tablet: real 0.051s	user 0.036s	sys 0.012s
I20240502 21:25:58.959920 31487 raft_consensus.cc:348] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: NON_VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: true } }
I20240502 21:25:58.960449 31487 raft_consensus.cc:729] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a9ca254f89984b3f9ed62020bd6a4d02, State: Initialized, Role: LEARNER
I20240502 21:25:58.961213 31487 consensus_queue.cc:260] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: NON_VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: true } }
I20240502 21:25:58.963289 31487 ts_tablet_manager.cc:1398] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02: Time spent starting tablet: real 0.009s	user 0.011s	sys 0.000s
I20240502 21:25:58.964615 29050 tablet_copy_service.cc:342] P 52c5becf6262415cab0408b3968ed4c0: Request end of tablet copy session a9ca254f89984b3f9ed62020bd6a4d02-876004fec43e428e930f3fbb873ccac8 received from {username='slave'} at 127.27.196.71:34711
I20240502 21:25:58.964869 29050 tablet_copy_service.cc:433] P 52c5becf6262415cab0408b3968ed4c0: ending tablet copy session a9ca254f89984b3f9ed62020bd6a4d02-876004fec43e428e930f3fbb873ccac8 on tablet 876004fec43e428e930f3fbb873ccac8 with peer a9ca254f89984b3f9ed62020bd6a4d02
W20240502 21:25:59.241084 28687 consensus_peers.cc:485] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 21:25:59.365425 30440 raft_consensus.cc:1203] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
W20240502 21:25:59.400085 28961 consensus_peers.cc:485] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:25:59.644413 28824 consensus_peers.cc:485] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 21:25:59.781245 31559 raft_consensus.cc:1050] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0: attempting to promote NON_VOTER a9ca254f89984b3f9ed62020bd6a4d02 to VOTER
I20240502 21:25:59.782286 31559 consensus_queue.cc:237] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } }
W20240502 21:25:59.785699 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 21:25:59.786002 30440 raft_consensus.cc:1261] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 LEARNER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:25:59.786000 28756 raft_consensus.cc:1261] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:25:59.786885 31559 consensus_queue.cc:1035] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:25:59.787358 31561 consensus_queue.cc:1035] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 21:25:59.787791 28961 consensus_peers.cc:485] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:59.791718 31559 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:25:59.792708 30440 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:25:59.795941 28756 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:25:59.797542 28471 catalog_manager.cc:5484] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 reported cstate change: config changed from index 5 to 6, a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "52c5becf6262415cab0408b3968ed4c0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:59.806385 29030 consensus_queue.cc:237] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } }
I20240502 21:25:59.808921 30440 raft_consensus.cc:1261] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:25:59.809530 28756 raft_consensus.cc:1261] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:25:59.810204 31558 consensus_queue.cc:1035] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:25:59.810578 31504 consensus_queue.cc:1035] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:25:59.813325 31561 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 04e36797791842e489aee30de261c4c7 (127.27.196.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:25:59.814025 28756 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 04e36797791842e489aee30de261c4c7 (127.27.196.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:25:59.816974 28462 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 876004fec43e428e930f3fbb873ccac8 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:25:59.817637 30440 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 04e36797791842e489aee30de261c4c7 (127.27.196.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:25:59.819591 28467 catalog_manager.cc:5484] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6 reported cstate change: config changed from index 6 to 7, VOTER 04e36797791842e489aee30de261c4c7 (127.27.196.65) evicted. New cstate: current_term: 1 leader_uuid: "52c5becf6262415cab0408b3968ed4c0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
W20240502 21:25:59.828315 28461 catalog_manager.cc:4573] TS 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 876004fec43e428e930f3fbb873ccac8: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
W20240502 21:25:59.967509 28687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 24 similar messages]
W20240502 21:25:59.968564 28687 consensus_peers.cc:485] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:26:00.004649 28824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 21:26:00.006357 28824 consensus_peers.cc:485] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:26:00.014377 28824 consensus_peers.cc:485] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:26:00.175637 28961 consensus_peers.cc:485] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 21:26:00.190534 30552 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:00.204239 29006 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:26:00.211138 28871 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:26:00.224555 30152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:00.226682 30285 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:00.240201 28735 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:26:00.259819 30420 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:00.467689 30552 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:00.490711 28871 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:26:00.492172 30420 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:00.507630 29006 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:26:00.527604 30152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:00.532852 30285 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:00.550233 28735 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:26:00.634822 28756 consensus_queue.cc:237] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } }
I20240502 21:26:00.638010 30305 raft_consensus.cc:1261] T 3665b7b9141b4fe8965eeecea35e9c0b P fc5aa2dc1f7b4554906785d693e272d0 [term 3 FOLLOWER]: Refusing update from remote peer 8786a1419d3f4bc79b88c54ce0675aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240502 21:26:00.638376 28893 raft_consensus.cc:1261] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 3 FOLLOWER]: Refusing update from remote peer 8786a1419d3f4bc79b88c54ce0675aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240502 21:26:00.638885 31575 consensus_queue.cc:1035] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:26:00.640830 31575 consensus_queue.cc:1035] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 21:26:00.654418 28686 consensus_peers.cc:485] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72:34997): Couldn't send request to peer 4e8bd1e7c1f848b0bdd2570de97506d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3665b7b9141b4fe8965eeecea35e9c0b. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:00.656008 28893 raft_consensus.cc:2932] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 8 to 9, NON_VOTER 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } } }
I20240502 21:26:00.650162 31575 raft_consensus.cc:2932] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 3 LEADER]: Committing config change with OpId 3.9: config changed from index 8 to 9, NON_VOTER 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } } }
I20240502 21:26:00.659296 30305 raft_consensus.cc:2932] T 3665b7b9141b4fe8965eeecea35e9c0b P fc5aa2dc1f7b4554906785d693e272d0 [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 8 to 9, NON_VOTER 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } } }
I20240502 21:26:00.664904 28475 catalog_manager.cc:5484] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 reported cstate change: config changed from index 8 to 9, NON_VOTER 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) added. New cstate: current_term: 3 leader_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } } }
I20240502 21:26:00.684494 28893 consensus_queue.cc:237] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: true } }
I20240502 21:26:00.688773 30305 raft_consensus.cc:1261] T 4a58681ae3e54019aefa430aaa3dc046 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:00.688773 29030 raft_consensus.cc:1261] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:00.689496 31639 consensus_queue.cc:1035] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:00.689930 31505 consensus_queue.cc:1035] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:00.693655 31505 raft_consensus.cc:2932] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: true } } }
W20240502 21:26:00.695457 28824 consensus_peers.cc:485] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 -> Peer a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69:41545): Couldn't send request to peer a1e2dba8e1a3403295cb394f7090f849. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a58681ae3e54019aefa430aaa3dc046. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:00.694983 30305 raft_consensus.cc:2932] T 4a58681ae3e54019aefa430aaa3dc046 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: true } } }
I20240502 21:26:00.698433 29030 raft_consensus.cc:2932] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: true } } }
I20240502 21:26:00.700485 28471 catalog_manager.cc:5484] T 4a58681ae3e54019aefa430aaa3dc046 P fc5aa2dc1f7b4554906785d693e272d0 reported cstate change: config changed from index 7 to 8, NON_VOTER a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) added. New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: true } } }
I20240502 21:26:00.720041 29030 consensus_queue.cc:237] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: NON_VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: true } }
I20240502 21:26:00.724186 28756 raft_consensus.cc:1261] T 5370969bf2f44ee0833fafa4285edf9f P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:00.724159 28893 raft_consensus.cc:1261] T 5370969bf2f44ee0833fafa4285edf9f P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:26:00.724661 28962 consensus_peers.cc:485] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 -> Peer a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71:36827): Couldn't send request to peer a9ca254f89984b3f9ed62020bd6a4d02. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5370969bf2f44ee0833fafa4285edf9f. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:00.725026 31561 consensus_queue.cc:1035] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:00.725723 31561 consensus_queue.cc:1035] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:00.736920 31561 raft_consensus.cc:2932] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: NON_VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: true } } }
I20240502 21:26:00.738883 28756 raft_consensus.cc:2932] T 5370969bf2f44ee0833fafa4285edf9f P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: NON_VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: true } } }
I20240502 21:26:00.738886 28893 raft_consensus.cc:2932] T 5370969bf2f44ee0833fafa4285edf9f P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: NON_VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: true } } }
I20240502 21:26:00.744195 28475 catalog_manager.cc:5484] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 reported cstate change: config changed from index -1 to 4, NON_VOTER a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) added. New cstate: current_term: 1 leader_uuid: "52c5becf6262415cab0408b3968ed4c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: NON_VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:26:00.998999 28461 catalog_manager.cc:4573] TS 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 876004fec43e428e930f3fbb873ccac8: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
I20240502 21:26:01.107570 31718 ts_tablet_manager.cc:902] T 3665b7b9141b4fe8965eeecea35e9c0b P 4e8bd1e7c1f848b0bdd2570de97506d8: Initiating tablet copy from peer 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815)
I20240502 21:26:01.109632 31718 tablet_copy_client.cc:308] T 3665b7b9141b4fe8965eeecea35e9c0b P 4e8bd1e7c1f848b0bdd2570de97506d8: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:41815
I20240502 21:26:01.117374 28777 tablet_copy_service.cc:140] P 8786a1419d3f4bc79b88c54ce0675aa6: Received BeginTabletCopySession request for tablet 3665b7b9141b4fe8965eeecea35e9c0b from peer 4e8bd1e7c1f848b0bdd2570de97506d8 ({username='slave'} at 127.27.196.72:52015)
I20240502 21:26:01.117568 28777 tablet_copy_service.cc:161] P 8786a1419d3f4bc79b88c54ce0675aa6: Beginning new tablet copy session on tablet 3665b7b9141b4fe8965eeecea35e9c0b from peer 4e8bd1e7c1f848b0bdd2570de97506d8 at {username='slave'} at 127.27.196.72:52015: session id = 4e8bd1e7c1f848b0bdd2570de97506d8-3665b7b9141b4fe8965eeecea35e9c0b
I20240502 21:26:01.119925 28777 tablet_copy_source_session.cc:205] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:01.122260 31718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3665b7b9141b4fe8965eeecea35e9c0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:01.131090 31718 tablet_copy_client.cc:783] T 3665b7b9141b4fe8965eeecea35e9c0b P 4e8bd1e7c1f848b0bdd2570de97506d8: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:01.131675 31718 tablet_copy_client.cc:647] T 3665b7b9141b4fe8965eeecea35e9c0b P 4e8bd1e7c1f848b0bdd2570de97506d8: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:01.134048 31722 tablet_copy_client.cc:1002] T 3665b7b9141b4fe8965eeecea35e9c0b P 4e8bd1e7c1f848b0bdd2570de97506d8: tablet copy: Injecting latency into file download: 1500
I20240502 21:26:01.280339 31740 ts_tablet_manager.cc:902] T 5370969bf2f44ee0833fafa4285edf9f P a9ca254f89984b3f9ed62020bd6a4d02: Initiating tablet copy from peer 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795)
I20240502 21:26:01.284047 31740 tablet_copy_client.cc:308] T 5370969bf2f44ee0833fafa4285edf9f P a9ca254f89984b3f9ed62020bd6a4d02: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.68:41795
I20240502 21:26:01.285295 29050 tablet_copy_service.cc:140] P 52c5becf6262415cab0408b3968ed4c0: Received BeginTabletCopySession request for tablet 5370969bf2f44ee0833fafa4285edf9f from peer a9ca254f89984b3f9ed62020bd6a4d02 ({username='slave'} at 127.27.196.71:34711)
I20240502 21:26:01.285527 29050 tablet_copy_service.cc:161] P 52c5becf6262415cab0408b3968ed4c0: Beginning new tablet copy session on tablet 5370969bf2f44ee0833fafa4285edf9f from peer a9ca254f89984b3f9ed62020bd6a4d02 at {username='slave'} at 127.27.196.71:34711: session id = a9ca254f89984b3f9ed62020bd6a4d02-5370969bf2f44ee0833fafa4285edf9f
I20240502 21:26:01.288297 29050 tablet_copy_source_session.cc:205] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:01.290005 31740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5370969bf2f44ee0833fafa4285edf9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:01.294684 31740 tablet_copy_client.cc:783] T 5370969bf2f44ee0833fafa4285edf9f P a9ca254f89984b3f9ed62020bd6a4d02: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:01.294955 31740 tablet_copy_client.cc:647] T 5370969bf2f44ee0833fafa4285edf9f P a9ca254f89984b3f9ed62020bd6a4d02: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:01.296938 31744 tablet_copy_client.cc:1002] T 5370969bf2f44ee0833fafa4285edf9f P a9ca254f89984b3f9ed62020bd6a4d02: tablet copy: Injecting latency into file download: 1500
I20240502 21:26:01.309779 31745 ts_tablet_manager.cc:902] T 4a58681ae3e54019aefa430aaa3dc046 P a1e2dba8e1a3403295cb394f7090f849: Initiating tablet copy from peer 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545)
I20240502 21:26:01.311065 31745 tablet_copy_client.cc:308] T 4a58681ae3e54019aefa430aaa3dc046 P a1e2dba8e1a3403295cb394f7090f849: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:42545
I20240502 21:26:01.318228 28913 tablet_copy_service.cc:140] P 8733381e64da4e91a8a54e23b01e8967: Received BeginTabletCopySession request for tablet 4a58681ae3e54019aefa430aaa3dc046 from peer a1e2dba8e1a3403295cb394f7090f849 ({username='slave'} at 127.27.196.69:38737)
I20240502 21:26:01.318457 28913 tablet_copy_service.cc:161] P 8733381e64da4e91a8a54e23b01e8967: Beginning new tablet copy session on tablet 4a58681ae3e54019aefa430aaa3dc046 from peer a1e2dba8e1a3403295cb394f7090f849 at {username='slave'} at 127.27.196.69:38737: session id = a1e2dba8e1a3403295cb394f7090f849-4a58681ae3e54019aefa430aaa3dc046
I20240502 21:26:01.320619 28913 tablet_copy_source_session.cc:205] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:01.322149 31745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a58681ae3e54019aefa430aaa3dc046. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:01.326961 31745 tablet_copy_client.cc:783] T 4a58681ae3e54019aefa430aaa3dc046 P a1e2dba8e1a3403295cb394f7090f849: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:01.327291 31745 tablet_copy_client.cc:647] T 4a58681ae3e54019aefa430aaa3dc046 P a1e2dba8e1a3403295cb394f7090f849: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:01.329568 31749 tablet_copy_client.cc:1002] T 4a58681ae3e54019aefa430aaa3dc046 P a1e2dba8e1a3403295cb394f7090f849: tablet copy: Injecting latency into file download: 1500
W20240502 21:26:01.411080 28961 consensus_peers.cc:485] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:26:01.944712 28687 consensus_peers.cc:485] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:26:02.063472 28461 catalog_manager.cc:4573] TS 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 876004fec43e428e930f3fbb873ccac8: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
W20240502 21:26:02.274624 28824 consensus_peers.cc:485] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:26:02.572369 28824 consensus_peers.cc:485] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:26:02.575608 28824 consensus_peers.cc:485] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 21:26:02.634683 31718 tablet_copy_client.cc:523] T 3665b7b9141b4fe8965eeecea35e9c0b P 4e8bd1e7c1f848b0bdd2570de97506d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:02.637941 31718 tablet_bootstrap.cc:492] T 3665b7b9141b4fe8965eeecea35e9c0b P 4e8bd1e7c1f848b0bdd2570de97506d8: Bootstrap starting.
I20240502 21:26:02.684340 31718 tablet_bootstrap.cc:492] T 3665b7b9141b4fe8965eeecea35e9c0b P 4e8bd1e7c1f848b0bdd2570de97506d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:02.684896 31718 tablet_bootstrap.cc:492] T 3665b7b9141b4fe8965eeecea35e9c0b P 4e8bd1e7c1f848b0bdd2570de97506d8: Bootstrap complete.
I20240502 21:26:02.685223 31718 ts_tablet_manager.cc:1367] T 3665b7b9141b4fe8965eeecea35e9c0b P 4e8bd1e7c1f848b0bdd2570de97506d8: Time spent bootstrapping tablet: real 0.047s	user 0.047s	sys 0.000s
I20240502 21:26:02.686205 31718 raft_consensus.cc:348] T 3665b7b9141b4fe8965eeecea35e9c0b P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } }
I20240502 21:26:02.686496 31718 raft_consensus.cc:729] T 3665b7b9141b4fe8965eeecea35e9c0b P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 4e8bd1e7c1f848b0bdd2570de97506d8, State: Initialized, Role: LEARNER
I20240502 21:26:02.686715 31718 consensus_queue.cc:260] T 3665b7b9141b4fe8965eeecea35e9c0b P 4e8bd1e7c1f848b0bdd2570de97506d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 3.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } }
I20240502 21:26:02.687852 31718 ts_tablet_manager.cc:1398] T 3665b7b9141b4fe8965eeecea35e9c0b P 4e8bd1e7c1f848b0bdd2570de97506d8: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.004s
I20240502 21:26:02.688901 28777 tablet_copy_service.cc:342] P 8786a1419d3f4bc79b88c54ce0675aa6: Request end of tablet copy session 4e8bd1e7c1f848b0bdd2570de97506d8-3665b7b9141b4fe8965eeecea35e9c0b received from {username='slave'} at 127.27.196.72:52015
I20240502 21:26:02.689106 28777 tablet_copy_service.cc:433] P 8786a1419d3f4bc79b88c54ce0675aa6: ending tablet copy session 4e8bd1e7c1f848b0bdd2570de97506d8-3665b7b9141b4fe8965eeecea35e9c0b on tablet 3665b7b9141b4fe8965eeecea35e9c0b with peer 4e8bd1e7c1f848b0bdd2570de97506d8
I20240502 21:26:02.690212 30572 raft_consensus.cc:1203] T 3665b7b9141b4fe8965eeecea35e9c0b P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.8->[3.9-3.9]   Dedup: 3.9->[]
W20240502 21:26:02.782936 28461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 14 similar messages]
I20240502 21:26:02.797490 31740 tablet_copy_client.cc:523] T 5370969bf2f44ee0833fafa4285edf9f P a9ca254f89984b3f9ed62020bd6a4d02: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:02.800500 31740 tablet_bootstrap.cc:492] T 5370969bf2f44ee0833fafa4285edf9f P a9ca254f89984b3f9ed62020bd6a4d02: Bootstrap starting.
I20240502 21:26:02.830152 31745 tablet_copy_client.cc:523] T 4a58681ae3e54019aefa430aaa3dc046 P a1e2dba8e1a3403295cb394f7090f849: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:02.833462 31745 tablet_bootstrap.cc:492] T 4a58681ae3e54019aefa430aaa3dc046 P a1e2dba8e1a3403295cb394f7090f849: Bootstrap starting.
I20240502 21:26:02.834857 31740 tablet_bootstrap.cc:492] T 5370969bf2f44ee0833fafa4285edf9f P a9ca254f89984b3f9ed62020bd6a4d02: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:02.835562 31740 tablet_bootstrap.cc:492] T 5370969bf2f44ee0833fafa4285edf9f P a9ca254f89984b3f9ed62020bd6a4d02: Bootstrap complete.
I20240502 21:26:02.835899 31740 ts_tablet_manager.cc:1367] T 5370969bf2f44ee0833fafa4285edf9f P a9ca254f89984b3f9ed62020bd6a4d02: Time spent bootstrapping tablet: real 0.035s	user 0.031s	sys 0.004s
I20240502 21:26:02.836974 31740 raft_consensus.cc:348] T 5370969bf2f44ee0833fafa4285edf9f P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: NON_VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: true } }
I20240502 21:26:02.837430 31740 raft_consensus.cc:729] T 5370969bf2f44ee0833fafa4285edf9f P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a9ca254f89984b3f9ed62020bd6a4d02, State: Initialized, Role: LEARNER
I20240502 21:26:02.837714 31740 consensus_queue.cc:260] T 5370969bf2f44ee0833fafa4285edf9f P a9ca254f89984b3f9ed62020bd6a4d02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: NON_VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: true } }
I20240502 21:26:02.838742 31740 ts_tablet_manager.cc:1398] T 5370969bf2f44ee0833fafa4285edf9f P a9ca254f89984b3f9ed62020bd6a4d02: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.001s
I20240502 21:26:02.839957 29050 tablet_copy_service.cc:342] P 52c5becf6262415cab0408b3968ed4c0: Request end of tablet copy session a9ca254f89984b3f9ed62020bd6a4d02-5370969bf2f44ee0833fafa4285edf9f received from {username='slave'} at 127.27.196.71:34711
I20240502 21:26:02.840152 29050 tablet_copy_service.cc:433] P 52c5becf6262415cab0408b3968ed4c0: ending tablet copy session a9ca254f89984b3f9ed62020bd6a4d02-5370969bf2f44ee0833fafa4285edf9f on tablet 5370969bf2f44ee0833fafa4285edf9f with peer a9ca254f89984b3f9ed62020bd6a4d02
W20240502 21:26:02.850335 28687 consensus_peers.cc:485] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 21:26:02.874584 31745 tablet_bootstrap.cc:492] T 4a58681ae3e54019aefa430aaa3dc046 P a1e2dba8e1a3403295cb394f7090f849: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:02.875177 31745 tablet_bootstrap.cc:492] T 4a58681ae3e54019aefa430aaa3dc046 P a1e2dba8e1a3403295cb394f7090f849: Bootstrap complete.
I20240502 21:26:02.875531 31745 ts_tablet_manager.cc:1367] T 4a58681ae3e54019aefa430aaa3dc046 P a1e2dba8e1a3403295cb394f7090f849: Time spent bootstrapping tablet: real 0.042s	user 0.039s	sys 0.004s
I20240502 21:26:02.876436 31745 raft_consensus.cc:348] T 4a58681ae3e54019aefa430aaa3dc046 P a1e2dba8e1a3403295cb394f7090f849 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: true } }
I20240502 21:26:02.876680 31745 raft_consensus.cc:729] T 4a58681ae3e54019aefa430aaa3dc046 P a1e2dba8e1a3403295cb394f7090f849 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a1e2dba8e1a3403295cb394f7090f849, State: Initialized, Role: LEARNER
I20240502 21:26:02.876900 31745 consensus_queue.cc:260] T 4a58681ae3e54019aefa430aaa3dc046 P a1e2dba8e1a3403295cb394f7090f849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: true } }
I20240502 21:26:02.877929 31745 ts_tablet_manager.cc:1398] T 4a58681ae3e54019aefa430aaa3dc046 P a1e2dba8e1a3403295cb394f7090f849: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:26:02.878935 28913 tablet_copy_service.cc:342] P 8733381e64da4e91a8a54e23b01e8967: Request end of tablet copy session a1e2dba8e1a3403295cb394f7090f849-4a58681ae3e54019aefa430aaa3dc046 received from {username='slave'} at 127.27.196.69:38737
I20240502 21:26:02.879168 28913 tablet_copy_service.cc:433] P 8733381e64da4e91a8a54e23b01e8967: ending tablet copy session a1e2dba8e1a3403295cb394f7090f849-4a58681ae3e54019aefa430aaa3dc046 on tablet 4a58681ae3e54019aefa430aaa3dc046 with peer a1e2dba8e1a3403295cb394f7090f849
W20240502 21:26:02.901826 28961 consensus_peers.cc:485] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 21:26:03.153631 30172 raft_consensus.cc:1203] T 4a58681ae3e54019aefa430aaa3dc046 P a1e2dba8e1a3403295cb394f7090f849 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 21:26:03.270015 31893 raft_consensus.cc:1050] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6: attempting to promote NON_VOTER 4e8bd1e7c1f848b0bdd2570de97506d8 to VOTER
I20240502 21:26:03.270994 31893 consensus_queue.cc:237] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } }
I20240502 21:26:03.275401 28893 raft_consensus.cc:1261] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 3 FOLLOWER]: Refusing update from remote peer 8786a1419d3f4bc79b88c54ce0675aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 21:26:03.275646 30572 raft_consensus.cc:1261] T 3665b7b9141b4fe8965eeecea35e9c0b P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 3 LEARNER]: Refusing update from remote peer 8786a1419d3f4bc79b88c54ce0675aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 21:26:03.276556 31837 consensus_queue.cc:1035] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:26:03.277392 30305 raft_consensus.cc:1261] T 3665b7b9141b4fe8965eeecea35e9c0b P fc5aa2dc1f7b4554906785d693e272d0 [term 3 FOLLOWER]: Refusing update from remote peer 8786a1419d3f4bc79b88c54ce0675aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 21:26:03.277367 31837 consensus_queue.cc:1035] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:26:03.278353 31837 consensus_queue.cc:1035] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:26:03.283756 31893 raft_consensus.cc:2932] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 3 LEADER]: Committing config change with OpId 3.10: config changed from index 9 to 10, 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:26:03.285130 28893 raft_consensus.cc:2932] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 3 FOLLOWER]: Committing config change with OpId 3.10: config changed from index 9 to 10, 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:26:03.285270 30305 raft_consensus.cc:2932] T 3665b7b9141b4fe8965eeecea35e9c0b P fc5aa2dc1f7b4554906785d693e272d0 [term 3 FOLLOWER]: Committing config change with OpId 3.10: config changed from index 9 to 10, 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:26:03.288913 30572 raft_consensus.cc:2932] T 3665b7b9141b4fe8965eeecea35e9c0b P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 3 FOLLOWER]: Committing config change with OpId 3.10: config changed from index 9 to 10, 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:26:03.290081 30440 raft_consensus.cc:1203] T 5370969bf2f44ee0833fafa4285edf9f P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:26:03.295603 28475 catalog_manager.cc:5484] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 reported cstate change: config changed from index 9 to 10, 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:26:03.308357 28756 consensus_queue.cc:237] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } }
I20240502 21:26:03.310986 30572 raft_consensus.cc:1261] T 3665b7b9141b4fe8965eeecea35e9c0b P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 3 FOLLOWER]: Refusing update from remote peer 8786a1419d3f4bc79b88c54ce0675aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 21:26:03.311184 30305 raft_consensus.cc:1261] T 3665b7b9141b4fe8965eeecea35e9c0b P fc5aa2dc1f7b4554906785d693e272d0 [term 3 FOLLOWER]: Refusing update from remote peer 8786a1419d3f4bc79b88c54ce0675aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 21:26:03.311728 31893 consensus_queue.cc:1035] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:26:03.312191 31896 consensus_queue.cc:1035] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:26:03.314970 31892 raft_consensus.cc:2932] T 3665b7b9141b4fe8965eeecea35e9c0b P 8786a1419d3f4bc79b88c54ce0675aa6 [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:26:03.315992 30572 raft_consensus.cc:2932] T 3665b7b9141b4fe8965eeecea35e9c0b P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:26:03.320050 28461 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3665b7b9141b4fe8965eeecea35e9c0b with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:03.319908 30305 raft_consensus.cc:2932] T 3665b7b9141b4fe8965eeecea35e9c0b P fc5aa2dc1f7b4554906785d693e272d0 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:26:03.322310 28472 catalog_manager.cc:5484] T 3665b7b9141b4fe8965eeecea35e9c0b P 4e8bd1e7c1f848b0bdd2570de97506d8 reported cstate change: config changed from index 10 to 11, VOTER 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67) evicted. New cstate: current_term: 3 leader_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:26:03.327944 28871 tablet_service.cc:1514] Processing DeleteTablet for tablet 3665b7b9141b4fe8965eeecea35e9c0b with delete_type TABLET_DATA_TOMBSTONED (TS 8733381e64da4e91a8a54e23b01e8967 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:53992
I20240502 21:26:03.329200 31906 tablet_replica.cc:330] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967: stopping tablet replica
I20240502 21:26:03.329826 31906 raft_consensus.cc:2229] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:03.330329 31906 raft_consensus.cc:2258] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:03.332599 31906 ts_tablet_manager.cc:1875] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:03.338040 31906 ts_tablet_manager.cc:1888] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.10
I20240502 21:26:03.338219 31906 log.cc:1196] T 3665b7b9141b4fe8965eeecea35e9c0b P 8733381e64da4e91a8a54e23b01e8967: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/wals/3665b7b9141b4fe8965eeecea35e9c0b
I20240502 21:26:03.339530 28460 catalog_manager.cc:4830] TS 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545): tablet 3665b7b9141b4fe8965eeecea35e9c0b (table rebalance_test_table_3 [id=e6d6bbf7aaec4d548ddf673c06866772]) successfully deleted
I20240502 21:26:03.604001 31788 raft_consensus.cc:1050] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967: attempting to promote NON_VOTER a1e2dba8e1a3403295cb394f7090f849 to VOTER
I20240502 21:26:03.604847 31788 consensus_queue.cc:237] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } }
I20240502 21:26:03.608268 30305 raft_consensus.cc:1261] T 4a58681ae3e54019aefa430aaa3dc046 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:26:03.608836 29029 raft_consensus.cc:1261] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:26:03.609122 31894 consensus_queue.cc:1035] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:26:03.610505 31894 consensus_queue.cc:1035] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:26:03.619184 30172 raft_consensus.cc:1261] T 4a58681ae3e54019aefa430aaa3dc046 P a1e2dba8e1a3403295cb394f7090f849 [term 1 LEARNER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:26:03.620263 31932 consensus_queue.cc:1035] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:26:03.625175 31894 raft_consensus.cc:2932] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
I20240502 21:26:03.627761 29029 raft_consensus.cc:2932] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
I20240502 21:26:03.636408 28472 catalog_manager.cc:5484] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0 reported cstate change: config changed from index 8 to 9, a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
I20240502 21:26:03.637045 30305 raft_consensus.cc:2932] T 4a58681ae3e54019aefa430aaa3dc046 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
I20240502 21:26:03.637080 30172 raft_consensus.cc:2932] T 4a58681ae3e54019aefa430aaa3dc046 P a1e2dba8e1a3403295cb394f7090f849 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
I20240502 21:26:03.643332 28893 consensus_queue.cc:237] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } }
I20240502 21:26:03.646402 30172 raft_consensus.cc:1261] T 4a58681ae3e54019aefa430aaa3dc046 P a1e2dba8e1a3403295cb394f7090f849 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:26:03.647092 31934 consensus_queue.cc:1035] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:26:03.648355 30304 raft_consensus.cc:1261] T 4a58681ae3e54019aefa430aaa3dc046 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:26:03.649915 31934 consensus_queue.cc:1035] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:26:03.652401 31932 raft_consensus.cc:2932] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
I20240502 21:26:03.653792 30304 raft_consensus.cc:2932] T 4a58681ae3e54019aefa430aaa3dc046 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
I20240502 21:26:03.658617 28460 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4a58681ae3e54019aefa430aaa3dc046 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:03.653215 30172 raft_consensus.cc:2932] T 4a58681ae3e54019aefa430aaa3dc046 P a1e2dba8e1a3403295cb394f7090f849 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
I20240502 21:26:03.661720 28471 catalog_manager.cc:5484] T 4a58681ae3e54019aefa430aaa3dc046 P 8733381e64da4e91a8a54e23b01e8967 reported cstate change: config changed from index 9 to 10, VOTER 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68) evicted. New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:03.667104 29006 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a58681ae3e54019aefa430aaa3dc046 with delete_type TABLET_DATA_TOMBSTONED (TS 52c5becf6262415cab0408b3968ed4c0 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:44172
I20240502 21:26:03.672926 31941 tablet_replica.cc:330] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0: stopping tablet replica
I20240502 21:26:03.673748 31941 raft_consensus.cc:2229] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:03.674263 31941 raft_consensus.cc:2258] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:03.676498 31941 ts_tablet_manager.cc:1875] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:03.691123 31941 ts_tablet_manager.cc:1888] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 21:26:03.692103 31943 raft_consensus.cc:1050] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0: attempting to promote NON_VOTER a9ca254f89984b3f9ed62020bd6a4d02 to VOTER
I20240502 21:26:03.693626 31943 consensus_queue.cc:237] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } }
I20240502 21:26:03.694317 31941 log.cc:1196] T 4a58681ae3e54019aefa430aaa3dc046 P 52c5becf6262415cab0408b3968ed4c0: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/wals/4a58681ae3e54019aefa430aaa3dc046
I20240502 21:26:03.696043 28462 catalog_manager.cc:4830] TS 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795): tablet 4a58681ae3e54019aefa430aaa3dc046 (table rebalance_test_table_2 [id=125451552e264fde833414d8cda8be18]) successfully deleted
I20240502 21:26:03.698493 28756 raft_consensus.cc:1261] T 5370969bf2f44ee0833fafa4285edf9f P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:03.698738 30440 raft_consensus.cc:1261] T 5370969bf2f44ee0833fafa4285edf9f P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 LEARNER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:03.699177 28893 raft_consensus.cc:1261] T 5370969bf2f44ee0833fafa4285edf9f P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:03.699267 31943 consensus_queue.cc:1035] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:03.700424 31860 consensus_queue.cc:1035] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:03.701032 31860 consensus_queue.cc:1035] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:03.716406 31860 raft_consensus.cc:2932] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:26:03.717365 30440 raft_consensus.cc:2932] T 5370969bf2f44ee0833fafa4285edf9f P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:26:03.722326 28893 raft_consensus.cc:2932] T 5370969bf2f44ee0833fafa4285edf9f P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:26:03.725868 28471 catalog_manager.cc:5484] T 5370969bf2f44ee0833fafa4285edf9f P a9ca254f89984b3f9ed62020bd6a4d02 reported cstate change: config changed from index 4 to 5, a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "52c5becf6262415cab0408b3968ed4c0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:26:03.736896 28756 raft_consensus.cc:2932] T 5370969bf2f44ee0833fafa4285edf9f P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
W20240502 21:26:03.747089 28961 consensus_peers.cc:485] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 21:26:03.758181 29029 consensus_queue.cc:237] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } }
I20240502 21:26:03.762635 30440 raft_consensus.cc:1261] T 5370969bf2f44ee0833fafa4285edf9f P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:03.764550 28893 raft_consensus.cc:1261] T 5370969bf2f44ee0833fafa4285edf9f P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:03.764569 31944 consensus_queue.cc:1035] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:03.767401 31944 consensus_queue.cc:1035] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:03.770138 30440 raft_consensus.cc:2932] T 5370969bf2f44ee0833fafa4285edf9f P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:26:03.772753 28893 raft_consensus.cc:2932] T 5370969bf2f44ee0833fafa4285edf9f P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:26:03.774294 31860 raft_consensus.cc:2932] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:26:03.780057 28462 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5370969bf2f44ee0833fafa4285edf9f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:03.781234 28472 catalog_manager.cc:5484] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 reported cstate change: config changed from index 5 to 6, VOTER 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66) evicted. New cstate: current_term: 1 leader_uuid: "52c5becf6262415cab0408b3968ed4c0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:03.788022 28735 tablet_service.cc:1514] Processing DeleteTablet for tablet 5370969bf2f44ee0833fafa4285edf9f with delete_type TABLET_DATA_TOMBSTONED (TS 8786a1419d3f4bc79b88c54ce0675aa6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34018
I20240502 21:26:03.795233 31956 tablet_replica.cc:330] T 5370969bf2f44ee0833fafa4285edf9f P 8786a1419d3f4bc79b88c54ce0675aa6: stopping tablet replica
I20240502 21:26:03.795910 31956 raft_consensus.cc:2229] T 5370969bf2f44ee0833fafa4285edf9f P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:03.796563 31956 raft_consensus.cc:2258] T 5370969bf2f44ee0833fafa4285edf9f P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:03.798472 31956 ts_tablet_manager.cc:1875] T 5370969bf2f44ee0833fafa4285edf9f P 8786a1419d3f4bc79b88c54ce0675aa6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:03.806686 31956 ts_tablet_manager.cc:1888] T 5370969bf2f44ee0833fafa4285edf9f P 8786a1419d3f4bc79b88c54ce0675aa6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:26:03.806879 31956 log.cc:1196] T 5370969bf2f44ee0833fafa4285edf9f P 8786a1419d3f4bc79b88c54ce0675aa6: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/wals/5370969bf2f44ee0833fafa4285edf9f
I20240502 21:26:03.808044 28461 catalog_manager.cc:4830] TS 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815): tablet 5370969bf2f44ee0833fafa4285edf9f (table rebalance_test_table_1 [id=a4cb53742e604e37ae1a6151409c162b]) successfully deleted
I20240502 21:26:03.823969 28893 consensus_queue.cc:237] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: true } }
I20240502 21:26:03.827400 28756 raft_consensus.cc:1261] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:03.827556 29029 raft_consensus.cc:1261] T c2cdd0cd6e344ca391d295f92c6f6658 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:03.828114 31788 consensus_queue.cc:1035] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:03.828768 31788 consensus_queue.cc:1035] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:26:03.829192 28824 consensus_peers.cc:485] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 -> Peer a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69:41545): Couldn't send request to peer a1e2dba8e1a3403295cb394f7090f849. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2cdd0cd6e344ca391d295f92c6f6658. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:03.831811 31788 raft_consensus.cc:2932] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: true } } }
I20240502 21:26:03.835611 28756 raft_consensus.cc:2932] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: true } } }
I20240502 21:26:03.839956 28471 catalog_manager.cc:5484] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 reported cstate change: config changed from index -1 to 4, NON_VOTER a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) added. New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:03.848868 29029 raft_consensus.cc:2932] T c2cdd0cd6e344ca391d295f92c6f6658 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: true } } }
I20240502 21:26:03.895767 28893 consensus_queue.cc:237] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } }
I20240502 21:26:03.898957 30572 raft_consensus.cc:1261] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:03.899570 28756 raft_consensus.cc:1261] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 21:26:03.899580 28826 consensus_peers.cc:485] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 -> Peer fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70:43969): Couldn't send request to peer fc5aa2dc1f7b4554906785d693e272d0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4c6f9c73a2d4b01a2c027504e45b238. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:03.900027 31932 consensus_queue.cc:1035] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:03.900467 31788 consensus_queue.cc:1035] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:03.904587 31934 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } } }
I20240502 21:26:03.909495 30572 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } } }
I20240502 21:26:03.910432 28756 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } } }
I20240502 21:26:03.914561 28476 catalog_manager.cc:5484] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 reported cstate change: config changed from index 7 to 8, NON_VOTER fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) added. New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:26:04.139616 28461 catalog_manager.cc:4573] TS 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 876004fec43e428e930f3fbb873ccac8: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
I20240502 21:26:04.223008 32007 ts_tablet_manager.cc:902] T c2cdd0cd6e344ca391d295f92c6f6658 P a1e2dba8e1a3403295cb394f7090f849: Initiating tablet copy from peer 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545)
I20240502 21:26:04.224859 32007 tablet_copy_client.cc:308] T c2cdd0cd6e344ca391d295f92c6f6658 P a1e2dba8e1a3403295cb394f7090f849: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:42545
I20240502 21:26:04.226102 28913 tablet_copy_service.cc:140] P 8733381e64da4e91a8a54e23b01e8967: Received BeginTabletCopySession request for tablet c2cdd0cd6e344ca391d295f92c6f6658 from peer a1e2dba8e1a3403295cb394f7090f849 ({username='slave'} at 127.27.196.69:38737)
I20240502 21:26:04.226389 28913 tablet_copy_service.cc:161] P 8733381e64da4e91a8a54e23b01e8967: Beginning new tablet copy session on tablet c2cdd0cd6e344ca391d295f92c6f6658 from peer a1e2dba8e1a3403295cb394f7090f849 at {username='slave'} at 127.27.196.69:38737: session id = a1e2dba8e1a3403295cb394f7090f849-c2cdd0cd6e344ca391d295f92c6f6658
I20240502 21:26:04.228665 28913 tablet_copy_source_session.cc:205] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:04.230729 32007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2cdd0cd6e344ca391d295f92c6f6658. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:04.238569 32007 tablet_copy_client.cc:783] T c2cdd0cd6e344ca391d295f92c6f6658 P a1e2dba8e1a3403295cb394f7090f849: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:04.238893 32007 tablet_copy_client.cc:647] T c2cdd0cd6e344ca391d295f92c6f6658 P a1e2dba8e1a3403295cb394f7090f849: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:04.241199 32011 tablet_copy_client.cc:1002] T c2cdd0cd6e344ca391d295f92c6f6658 P a1e2dba8e1a3403295cb394f7090f849: tablet copy: Injecting latency into file download: 1500
W20240502 21:26:04.475389 28687 consensus_peers.cc:485] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 21:26:04.502347 32022 ts_tablet_manager.cc:902] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0: Initiating tablet copy from peer 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545)
I20240502 21:26:04.503945 32022 tablet_copy_client.cc:308] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:42545
I20240502 21:26:04.505249 28913 tablet_copy_service.cc:140] P 8733381e64da4e91a8a54e23b01e8967: Received BeginTabletCopySession request for tablet a4c6f9c73a2d4b01a2c027504e45b238 from peer fc5aa2dc1f7b4554906785d693e272d0 ({username='slave'} at 127.27.196.70:40429)
I20240502 21:26:04.505427 28913 tablet_copy_service.cc:161] P 8733381e64da4e91a8a54e23b01e8967: Beginning new tablet copy session on tablet a4c6f9c73a2d4b01a2c027504e45b238 from peer fc5aa2dc1f7b4554906785d693e272d0 at {username='slave'} at 127.27.196.70:40429: session id = fc5aa2dc1f7b4554906785d693e272d0-a4c6f9c73a2d4b01a2c027504e45b238
I20240502 21:26:04.507849 28913 tablet_copy_source_session.cc:205] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:04.509833 32022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c6f9c73a2d4b01a2c027504e45b238. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:04.514662 32022 tablet_copy_client.cc:783] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:04.514958 32022 tablet_copy_client.cc:647] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:04.517220 32026 tablet_copy_client.cc:1002] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0: tablet copy: Injecting latency into file download: 1500
W20240502 21:26:04.760483 28824 consensus_peers.cc:485] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:26:04.809059 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:26:05.175505 28824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 21:26:05.192101 28687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:26:05.193274 28687 consensus_peers.cc:485] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:26:05.210320 28824 consensus_peers.cc:485] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:26:05.369999 28824 consensus_peers.cc:485] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:26:05.421455 28961 consensus_peers.cc:485] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 21:26:05.742025 32007 tablet_copy_client.cc:523] T c2cdd0cd6e344ca391d295f92c6f6658 P a1e2dba8e1a3403295cb394f7090f849: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:05.746090 32007 tablet_bootstrap.cc:492] T c2cdd0cd6e344ca391d295f92c6f6658 P a1e2dba8e1a3403295cb394f7090f849: Bootstrap starting.
I20240502 21:26:05.788448 32007 tablet_bootstrap.cc:492] T c2cdd0cd6e344ca391d295f92c6f6658 P a1e2dba8e1a3403295cb394f7090f849: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:05.789168 32007 tablet_bootstrap.cc:492] T c2cdd0cd6e344ca391d295f92c6f6658 P a1e2dba8e1a3403295cb394f7090f849: Bootstrap complete.
I20240502 21:26:05.789769 32007 ts_tablet_manager.cc:1367] T c2cdd0cd6e344ca391d295f92c6f6658 P a1e2dba8e1a3403295cb394f7090f849: Time spent bootstrapping tablet: real 0.044s	user 0.041s	sys 0.004s
I20240502 21:26:05.790784 32007 raft_consensus.cc:348] T c2cdd0cd6e344ca391d295f92c6f6658 P a1e2dba8e1a3403295cb394f7090f849 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: true } }
I20240502 21:26:05.791142 32007 raft_consensus.cc:729] T c2cdd0cd6e344ca391d295f92c6f6658 P a1e2dba8e1a3403295cb394f7090f849 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a1e2dba8e1a3403295cb394f7090f849, State: Initialized, Role: LEARNER
I20240502 21:26:05.791474 32007 consensus_queue.cc:260] T c2cdd0cd6e344ca391d295f92c6f6658 P a1e2dba8e1a3403295cb394f7090f849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: true } }
I20240502 21:26:05.792393 32007 ts_tablet_manager.cc:1398] T c2cdd0cd6e344ca391d295f92c6f6658 P a1e2dba8e1a3403295cb394f7090f849: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.003s
I20240502 21:26:05.793233 28913 tablet_copy_service.cc:342] P 8733381e64da4e91a8a54e23b01e8967: Request end of tablet copy session a1e2dba8e1a3403295cb394f7090f849-c2cdd0cd6e344ca391d295f92c6f6658 received from {username='slave'} at 127.27.196.69:38737
I20240502 21:26:05.793387 28913 tablet_copy_service.cc:433] P 8733381e64da4e91a8a54e23b01e8967: ending tablet copy session a1e2dba8e1a3403295cb394f7090f849-c2cdd0cd6e344ca391d295f92c6f6658 on tablet c2cdd0cd6e344ca391d295f92c6f6658 with peer a1e2dba8e1a3403295cb394f7090f849
I20240502 21:26:05.812580 30172 raft_consensus.cc:1203] T c2cdd0cd6e344ca391d295f92c6f6658 P a1e2dba8e1a3403295cb394f7090f849 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:26:06.017764 32022 tablet_copy_client.cc:523] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:06.022104 32022 tablet_bootstrap.cc:492] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0: Bootstrap starting.
I20240502 21:26:06.059201 32022 tablet_bootstrap.cc:492] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:06.059674 32022 tablet_bootstrap.cc:492] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0: Bootstrap complete.
I20240502 21:26:06.059958 32022 ts_tablet_manager.cc:1367] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0: Time spent bootstrapping tablet: real 0.038s	user 0.035s	sys 0.004s
I20240502 21:26:06.060683 32022 raft_consensus.cc:348] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } }
I20240502 21:26:06.060907 32022 raft_consensus.cc:729] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fc5aa2dc1f7b4554906785d693e272d0, State: Initialized, Role: LEARNER
I20240502 21:26:06.061116 32022 consensus_queue.cc:260] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } }
I20240502 21:26:06.062047 32022 ts_tablet_manager.cc:1398] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:26:06.063228 28913 tablet_copy_service.cc:342] P 8733381e64da4e91a8a54e23b01e8967: Request end of tablet copy session fc5aa2dc1f7b4554906785d693e272d0-a4c6f9c73a2d4b01a2c027504e45b238 received from {username='slave'} at 127.27.196.70:40429
I20240502 21:26:06.063366 28913 tablet_copy_service.cc:433] P 8733381e64da4e91a8a54e23b01e8967: ending tablet copy session fc5aa2dc1f7b4554906785d693e272d0-a4c6f9c73a2d4b01a2c027504e45b238 on tablet a4c6f9c73a2d4b01a2c027504e45b238 with peer fc5aa2dc1f7b4554906785d693e272d0
W20240502 21:26:06.107754 28961 consensus_peers.cc:485] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 21:26:06.333411 32118 raft_consensus.cc:1050] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967: attempting to promote NON_VOTER a1e2dba8e1a3403295cb394f7090f849 to VOTER
I20240502 21:26:06.334761 32118 consensus_queue.cc:237] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } }
I20240502 21:26:06.338254 29029 raft_consensus.cc:1261] T c2cdd0cd6e344ca391d295f92c6f6658 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:06.338682 28756 raft_consensus.cc:1261] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:06.339140 30172 raft_consensus.cc:1261] T c2cdd0cd6e344ca391d295f92c6f6658 P a1e2dba8e1a3403295cb394f7090f849 [term 1 LEARNER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:06.339499 32120 consensus_queue.cc:1035] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:06.341030 32118 consensus_queue.cc:1035] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:06.341564 32028 consensus_queue.cc:1035] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:06.346649 32120 raft_consensus.cc:2932] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
I20240502 21:26:06.347884 29029 raft_consensus.cc:2932] T c2cdd0cd6e344ca391d295f92c6f6658 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
I20240502 21:26:06.349012 30172 raft_consensus.cc:2932] T c2cdd0cd6e344ca391d295f92c6f6658 P a1e2dba8e1a3403295cb394f7090f849 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
I20240502 21:26:06.354195 28471 catalog_manager.cc:5484] T c2cdd0cd6e344ca391d295f92c6f6658 P 52c5becf6262415cab0408b3968ed4c0 reported cstate change: config changed from index 4 to 5, a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
I20240502 21:26:06.358824 28756 raft_consensus.cc:2932] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
I20240502 21:26:06.376029 28893 consensus_queue.cc:237] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } }
I20240502 21:26:06.378841 28756 raft_consensus.cc:1261] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:06.378829 30172 raft_consensus.cc:1261] T c2cdd0cd6e344ca391d295f92c6f6658 P a1e2dba8e1a3403295cb394f7090f849 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:06.379942 32120 consensus_queue.cc:1035] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:06.380455 32118 consensus_queue.cc:1035] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:06.384562 32120 raft_consensus.cc:2932] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
I20240502 21:26:06.386173 28756 raft_consensus.cc:2932] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
I20240502 21:26:06.390920 28460 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c2cdd0cd6e344ca391d295f92c6f6658 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:06.397621 28476 catalog_manager.cc:5484] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 reported cstate change: config changed from index 5 to 6, VOTER 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68) evicted. New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
I20240502 21:26:06.398497 30172 raft_consensus.cc:2932] T c2cdd0cd6e344ca391d295f92c6f6658 P a1e2dba8e1a3403295cb394f7090f849 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
I20240502 21:26:06.404377 29006 tablet_service.cc:1514] Processing DeleteTablet for tablet c2cdd0cd6e344ca391d295f92c6f6658 with delete_type TABLET_DATA_TOMBSTONED (TS 52c5becf6262415cab0408b3968ed4c0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44172
W20240502 21:26:06.406466 28461 catalog_manager.cc:4573] TS 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a2fd105773d449da89940d3b6558361a: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
I20240502 21:26:06.406724 32131 tablet_replica.cc:330] T c2cdd0cd6e344ca391d295f92c6f6658 P 52c5becf6262415cab0408b3968ed4c0: stopping tablet replica
I20240502 21:26:06.407320 32131 raft_consensus.cc:2229] T c2cdd0cd6e344ca391d295f92c6f6658 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:06.407743 32131 raft_consensus.cc:2258] T c2cdd0cd6e344ca391d295f92c6f6658 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:06.409261 32131 ts_tablet_manager.cc:1875] T c2cdd0cd6e344ca391d295f92c6f6658 P 52c5becf6262415cab0408b3968ed4c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:06.415558 32131 ts_tablet_manager.cc:1888] T c2cdd0cd6e344ca391d295f92c6f6658 P 52c5becf6262415cab0408b3968ed4c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:26:06.415738 32131 log.cc:1196] T c2cdd0cd6e344ca391d295f92c6f6658 P 52c5becf6262415cab0408b3968ed4c0: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/wals/c2cdd0cd6e344ca391d295f92c6f6658
I20240502 21:26:06.417018 28462 catalog_manager.cc:4830] TS 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795): tablet c2cdd0cd6e344ca391d295f92c6f6658 (table rebalance_test_table_1 [id=a4cb53742e604e37ae1a6151409c162b]) successfully deleted
I20240502 21:26:06.434264 30304 raft_consensus.cc:1203] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 21:26:06.494647 28893 consensus_queue.cc:237] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: NON_VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: true } }
I20240502 21:26:06.497473 28756 raft_consensus.cc:1261] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:06.497750 29029 raft_consensus.cc:1261] T 2c630e4bed554c8fa99ecfabed6519f1 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:06.498216 32120 consensus_queue.cc:1035] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:06.498589 32118 consensus_queue.cc:1035] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:06.502955 32118 raft_consensus.cc:2932] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: NON_VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: true } } }
I20240502 21:26:06.503995 29029 raft_consensus.cc:2932] T 2c630e4bed554c8fa99ecfabed6519f1 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: NON_VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: true } } }
I20240502 21:26:06.505656 28756 raft_consensus.cc:2932] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: NON_VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: true } } }
I20240502 21:26:06.509924 28476 catalog_manager.cc:5484] T 2c630e4bed554c8fa99ecfabed6519f1 P 52c5becf6262415cab0408b3968ed4c0 reported cstate change: config changed from index -1 to 4, NON_VOTER a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) added. New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: NON_VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: true } } }
W20240502 21:26:06.512707 28825 consensus_peers.cc:485] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 -> Peer a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71:36827): Couldn't send request to peer a9ca254f89984b3f9ed62020bd6a4d02. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c630e4bed554c8fa99ecfabed6519f1. This is attempt 1: this message will repeat every 5th retry.
W20240502 21:26:06.911473 28687 consensus_peers.cc:485] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 21:26:06.917992 32118 raft_consensus.cc:1050] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967: attempting to promote NON_VOTER fc5aa2dc1f7b4554906785d693e272d0 to VOTER
I20240502 21:26:06.918959 32118 consensus_queue.cc:237] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } }
I20240502 21:26:06.922297 30304 raft_consensus.cc:1261] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 LEARNER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:26:06.922219 28756 raft_consensus.cc:1261] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:26:06.922940 30572 raft_consensus.cc:1261] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:26:06.923298 32173 consensus_queue.cc:1035] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:26:06.923882 32118 consensus_queue.cc:1035] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:26:06.924235 32028 consensus_queue.cc:1035] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:26:06.928562 32173 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:26:06.929567 30304 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:26:06.930128 28755 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:26:06.934702 30572 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:26:06.936269 28475 catalog_manager.cc:5484] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6 reported cstate change: config changed from index 8 to 9, fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:26:06.943609 28893 consensus_queue.cc:237] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } }
I20240502 21:26:06.950801 30304 raft_consensus.cc:1261] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:26:06.952493 32173 consensus_queue.cc:1035] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:26:06.954641 30571 raft_consensus.cc:1261] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:26:06.956068 32120 consensus_queue.cc:1035] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:26:06.957707 30304 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:26:06.958202 30571 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:26:06.963069 32118 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:26:06.966022 28471 catalog_manager.cc:5484] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0 reported cstate change: config changed from index 9 to 10, VOTER 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66) evicted. New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:26:06.969866 28460 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a4c6f9c73a2d4b01a2c027504e45b238 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:06.970602 28735 tablet_service.cc:1514] Processing DeleteTablet for tablet a4c6f9c73a2d4b01a2c027504e45b238 with delete_type TABLET_DATA_TOMBSTONED (TS 8786a1419d3f4bc79b88c54ce0675aa6 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:34018
I20240502 21:26:06.972396 32178 tablet_replica.cc:330] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6: stopping tablet replica
I20240502 21:26:06.972622 32178 raft_consensus.cc:2229] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:06.973045 32178 raft_consensus.cc:2258] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:06.975522 32178 ts_tablet_manager.cc:1875] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:06.982992 32178 ts_tablet_manager.cc:1888] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 21:26:06.983182 32178 log.cc:1196] T a4c6f9c73a2d4b01a2c027504e45b238 P 8786a1419d3f4bc79b88c54ce0675aa6: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/wals/a4c6f9c73a2d4b01a2c027504e45b238
I20240502 21:26:06.984730 28461 catalog_manager.cc:4830] TS 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815): tablet a4c6f9c73a2d4b01a2c027504e45b238 (table rebalance_test_table_4 [id=9474d8b6526a4d809c3b7e67f807f596]) successfully deleted
I20240502 21:26:07.030844 29029 consensus_queue.cc:237] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } }
I20240502 21:26:07.033097 32196 ts_tablet_manager.cc:902] T 2c630e4bed554c8fa99ecfabed6519f1 P a9ca254f89984b3f9ed62020bd6a4d02: Initiating tablet copy from peer 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545)
I20240502 21:26:07.035413 30440 raft_consensus.cc:1261] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:07.036589 31860 consensus_queue.cc:1035] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:07.038698 32196 tablet_copy_client.cc:308] T 2c630e4bed554c8fa99ecfabed6519f1 P a9ca254f89984b3f9ed62020bd6a4d02: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:42545
I20240502 21:26:07.039359 28755 raft_consensus.cc:1261] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:07.040432 32197 consensus_queue.cc:1035] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 21:26:07.047844 28960 consensus_peers.cc:485] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 -> Peer 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72:34997): Couldn't send request to peer 4e8bd1e7c1f848b0bdd2570de97506d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 876004fec43e428e930f3fbb873ccac8. This is attempt 1: this message will repeat every 5th retry.
W20240502 21:26:07.050403 28824 consensus_peers.cc:485] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 21:26:07.053939 31860 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } } }
I20240502 21:26:07.054808 28755 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } } }
I20240502 21:26:07.060518 28471 catalog_manager.cc:5484] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6 reported cstate change: config changed from index 7 to 8, NON_VOTER 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) added. New cstate: current_term: 1 leader_uuid: "52c5becf6262415cab0408b3968ed4c0" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } } }
I20240502 21:26:07.062399 28913 tablet_copy_service.cc:140] P 8733381e64da4e91a8a54e23b01e8967: Received BeginTabletCopySession request for tablet 2c630e4bed554c8fa99ecfabed6519f1 from peer a9ca254f89984b3f9ed62020bd6a4d02 ({username='slave'} at 127.27.196.71:41703)
I20240502 21:26:07.062611 28913 tablet_copy_service.cc:161] P 8733381e64da4e91a8a54e23b01e8967: Beginning new tablet copy session on tablet 2c630e4bed554c8fa99ecfabed6519f1 from peer a9ca254f89984b3f9ed62020bd6a4d02 at {username='slave'} at 127.27.196.71:41703: session id = a9ca254f89984b3f9ed62020bd6a4d02-2c630e4bed554c8fa99ecfabed6519f1
I20240502 21:26:07.065739 28913 tablet_copy_source_session.cc:205] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:07.067672 30440 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } } }
I20240502 21:26:07.069748 32196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c630e4bed554c8fa99ecfabed6519f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:07.078083 32196 tablet_copy_client.cc:783] T 2c630e4bed554c8fa99ecfabed6519f1 P a9ca254f89984b3f9ed62020bd6a4d02: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:07.078411 32196 tablet_copy_client.cc:647] T 2c630e4bed554c8fa99ecfabed6519f1 P a9ca254f89984b3f9ed62020bd6a4d02: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:07.080265 32202 tablet_copy_client.cc:1002] T 2c630e4bed554c8fa99ecfabed6519f1 P a9ca254f89984b3f9ed62020bd6a4d02: tablet copy: Injecting latency into file download: 1500
I20240502 21:26:07.598348 32239 ts_tablet_manager.cc:902] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8: Initiating tablet copy from peer 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795)
I20240502 21:26:07.600684 32239 tablet_copy_client.cc:308] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.68:41795
I20240502 21:26:07.608188 29050 tablet_copy_service.cc:140] P 52c5becf6262415cab0408b3968ed4c0: Received BeginTabletCopySession request for tablet 876004fec43e428e930f3fbb873ccac8 from peer 4e8bd1e7c1f848b0bdd2570de97506d8 ({username='slave'} at 127.27.196.72:41363)
I20240502 21:26:07.608469 29050 tablet_copy_service.cc:161] P 52c5becf6262415cab0408b3968ed4c0: Beginning new tablet copy session on tablet 876004fec43e428e930f3fbb873ccac8 from peer 4e8bd1e7c1f848b0bdd2570de97506d8 at {username='slave'} at 127.27.196.72:41363: session id = 4e8bd1e7c1f848b0bdd2570de97506d8-876004fec43e428e930f3fbb873ccac8
I20240502 21:26:07.613286 29050 tablet_copy_source_session.cc:205] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:07.615625 32239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 876004fec43e428e930f3fbb873ccac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:07.621938 32239 tablet_copy_client.cc:783] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:07.622246 32239 tablet_copy_client.cc:647] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:07.624529 32243 tablet_copy_client.cc:1002] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8: tablet copy: Injecting latency into file download: 1500
W20240502 21:26:07.739956 28687 consensus_peers.cc:485] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:26:07.835651 28824 consensus_peers.cc:485] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:26:07.910641 28961 consensus_peers.cc:485] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:26:07.962577 28824 consensus_peers.cc:485] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:26:08.257584 28461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240502 21:26:08.259450 28461 catalog_manager.cc:4573] TS 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 876004fec43e428e930f3fbb873ccac8: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
I20240502 21:26:08.580843 32196 tablet_copy_client.cc:523] T 2c630e4bed554c8fa99ecfabed6519f1 P a9ca254f89984b3f9ed62020bd6a4d02: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:08.584880 32196 tablet_bootstrap.cc:492] T 2c630e4bed554c8fa99ecfabed6519f1 P a9ca254f89984b3f9ed62020bd6a4d02: Bootstrap starting.
W20240502 21:26:08.602134 28961 consensus_peers.cc:485] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 21:26:08.620054 32196 tablet_bootstrap.cc:492] T 2c630e4bed554c8fa99ecfabed6519f1 P a9ca254f89984b3f9ed62020bd6a4d02: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:08.620554 32196 tablet_bootstrap.cc:492] T 2c630e4bed554c8fa99ecfabed6519f1 P a9ca254f89984b3f9ed62020bd6a4d02: Bootstrap complete.
I20240502 21:26:08.620898 32196 ts_tablet_manager.cc:1367] T 2c630e4bed554c8fa99ecfabed6519f1 P a9ca254f89984b3f9ed62020bd6a4d02: Time spent bootstrapping tablet: real 0.036s	user 0.036s	sys 0.000s
I20240502 21:26:08.621631 32196 raft_consensus.cc:348] T 2c630e4bed554c8fa99ecfabed6519f1 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: NON_VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: true } }
I20240502 21:26:08.621870 32196 raft_consensus.cc:729] T 2c630e4bed554c8fa99ecfabed6519f1 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a9ca254f89984b3f9ed62020bd6a4d02, State: Initialized, Role: LEARNER
I20240502 21:26:08.622045 32196 consensus_queue.cc:260] T 2c630e4bed554c8fa99ecfabed6519f1 P a9ca254f89984b3f9ed62020bd6a4d02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: NON_VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: true } }
I20240502 21:26:08.622824 32196 ts_tablet_manager.cc:1398] T 2c630e4bed554c8fa99ecfabed6519f1 P a9ca254f89984b3f9ed62020bd6a4d02: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:26:08.624171 28913 tablet_copy_service.cc:342] P 8733381e64da4e91a8a54e23b01e8967: Request end of tablet copy session a9ca254f89984b3f9ed62020bd6a4d02-2c630e4bed554c8fa99ecfabed6519f1 received from {username='slave'} at 127.27.196.71:41703
I20240502 21:26:08.624342 28913 tablet_copy_service.cc:433] P 8733381e64da4e91a8a54e23b01e8967: ending tablet copy session a9ca254f89984b3f9ed62020bd6a4d02-2c630e4bed554c8fa99ecfabed6519f1 on tablet 2c630e4bed554c8fa99ecfabed6519f1 with peer a9ca254f89984b3f9ed62020bd6a4d02
I20240502 21:26:09.019960 30440 raft_consensus.cc:1203] T 2c630e4bed554c8fa99ecfabed6519f1 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:26:09.125138 32239 tablet_copy_client.cc:523] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:09.129200 32239 tablet_bootstrap.cc:492] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8: Bootstrap starting.
I20240502 21:26:09.166603 32239 tablet_bootstrap.cc:492] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:09.167038 32239 tablet_bootstrap.cc:492] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8: Bootstrap complete.
I20240502 21:26:09.167340 32239 ts_tablet_manager.cc:1367] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8: Time spent bootstrapping tablet: real 0.038s	user 0.031s	sys 0.008s
I20240502 21:26:09.168078 32239 raft_consensus.cc:348] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } }
I20240502 21:26:09.168294 32239 raft_consensus.cc:729] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e8bd1e7c1f848b0bdd2570de97506d8, State: Initialized, Role: LEARNER
I20240502 21:26:09.168453 32239 consensus_queue.cc:260] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } }
I20240502 21:26:09.169320 32239 ts_tablet_manager.cc:1398] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 21:26:09.170341 29050 tablet_copy_service.cc:342] P 52c5becf6262415cab0408b3968ed4c0: Request end of tablet copy session 4e8bd1e7c1f848b0bdd2570de97506d8-876004fec43e428e930f3fbb873ccac8 received from {username='slave'} at 127.27.196.72:41363
I20240502 21:26:09.170534 29050 tablet_copy_service.cc:433] P 52c5becf6262415cab0408b3968ed4c0: ending tablet copy session 4e8bd1e7c1f848b0bdd2570de97506d8-876004fec43e428e930f3fbb873ccac8 on tablet 876004fec43e428e930f3fbb873ccac8 with peer 4e8bd1e7c1f848b0bdd2570de97506d8
W20240502 21:26:09.435492 28687 consensus_peers.cc:485] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 21:26:09.550508 30571 raft_consensus.cc:1203] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 21:26:09.613636 32347 raft_consensus.cc:1050] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967: attempting to promote NON_VOTER a9ca254f89984b3f9ed62020bd6a4d02 to VOTER
I20240502 21:26:09.614652 32347 consensus_queue.cc:237] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } }
I20240502 21:26:09.617683 29029 raft_consensus.cc:1261] T 2c630e4bed554c8fa99ecfabed6519f1 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:09.617547 30440 raft_consensus.cc:1261] T 2c630e4bed554c8fa99ecfabed6519f1 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 LEARNER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:09.617678 28755 raft_consensus.cc:1261] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:09.618566 32347 consensus_queue.cc:1035] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:09.619077 32346 consensus_queue.cc:1035] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:09.619614 32349 consensus_queue.cc:1035] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:09.632494 32347 raft_consensus.cc:2932] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:26:09.633356 29029 raft_consensus.cc:2932] T 2c630e4bed554c8fa99ecfabed6519f1 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:26:09.633991 30440 raft_consensus.cc:2932] T 2c630e4bed554c8fa99ecfabed6519f1 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:26:09.638089 28755 raft_consensus.cc:2932] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:26:09.639910 28471 catalog_manager.cc:5484] T 2c630e4bed554c8fa99ecfabed6519f1 P 52c5becf6262415cab0408b3968ed4c0 reported cstate change: config changed from index 4 to 5, a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:26:09.660598 28893 consensus_queue.cc:237] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } }
I20240502 21:26:09.662884 30440 raft_consensus.cc:1261] T 2c630e4bed554c8fa99ecfabed6519f1 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:09.663015 28755 raft_consensus.cc:1261] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:09.663530 32354 consensus_queue.cc:1035] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:09.663923 32349 consensus_queue.cc:1035] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:09.667167 32349 raft_consensus.cc:2932] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:26:09.668061 30440 raft_consensus.cc:2932] T 2c630e4bed554c8fa99ecfabed6519f1 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:26:09.667995 28755 raft_consensus.cc:2932] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:26:09.671411 28460 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2c630e4bed554c8fa99ecfabed6519f1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:09.673136 28475 catalog_manager.cc:5484] T 2c630e4bed554c8fa99ecfabed6519f1 P a9ca254f89984b3f9ed62020bd6a4d02 reported cstate change: config changed from index 5 to 6, VOTER 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68) evicted. New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:26:09.677315 29006 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c630e4bed554c8fa99ecfabed6519f1 with delete_type TABLET_DATA_TOMBSTONED (TS 52c5becf6262415cab0408b3968ed4c0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44172
I20240502 21:26:09.678159 32360 tablet_replica.cc:330] T 2c630e4bed554c8fa99ecfabed6519f1 P 52c5becf6262415cab0408b3968ed4c0: stopping tablet replica
I20240502 21:26:09.678588 32360 raft_consensus.cc:2229] T 2c630e4bed554c8fa99ecfabed6519f1 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:09.679231 32360 raft_consensus.cc:2258] T 2c630e4bed554c8fa99ecfabed6519f1 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:09.680270 32360 ts_tablet_manager.cc:1875] T 2c630e4bed554c8fa99ecfabed6519f1 P 52c5becf6262415cab0408b3968ed4c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:09.687011 32360 ts_tablet_manager.cc:1888] T 2c630e4bed554c8fa99ecfabed6519f1 P 52c5becf6262415cab0408b3968ed4c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:26:09.687320 32360 log.cc:1196] T 2c630e4bed554c8fa99ecfabed6519f1 P 52c5becf6262415cab0408b3968ed4c0: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/wals/2c630e4bed554c8fa99ecfabed6519f1
I20240502 21:26:09.688495 28462 catalog_manager.cc:4830] TS 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795): tablet 2c630e4bed554c8fa99ecfabed6519f1 (table rebalance_test_table_3 [id=e6d6bbf7aaec4d548ddf673c06866772]) successfully deleted
W20240502 21:26:09.710762 28824 consensus_peers.cc:485] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:26:09.830346 28461 catalog_manager.cc:4573] TS 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4a58681ae3e54019aefa430aaa3dc046: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
W20240502 21:26:09.865243 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:26:10.042309 28687 consensus_peers.cc:485] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 21:26:10.156013 32386 raft_consensus.cc:1050] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0: attempting to promote NON_VOTER 4e8bd1e7c1f848b0bdd2570de97506d8 to VOTER
I20240502 21:26:10.156790 32386 consensus_queue.cc:237] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } }
I20240502 21:26:10.159829 30571 raft_consensus.cc:1261] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 LEARNER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:26:10.160008 28755 raft_consensus.cc:1261] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:26:10.160847 30440 raft_consensus.cc:1261] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:26:10.160722 32367 consensus_queue.cc:1035] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:26:10.161520 32266 consensus_queue.cc:1035] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:26:10.162261 32367 consensus_queue.cc:1035] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:26:10.165829 32388 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:26:10.166718 28755 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:26:10.167104 30440 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:26:10.171969 28472 catalog_manager.cc:5484] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 reported cstate change: config changed from index 8 to 9, 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "52c5becf6262415cab0408b3968ed4c0" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:10.172104 30571 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:26:10.180116 29029 consensus_queue.cc:237] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } }
I20240502 21:26:10.182397 30571 raft_consensus.cc:1261] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 FOLLOWER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:26:10.182418 30440 raft_consensus.cc:1261] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:26:10.183101 32266 consensus_queue.cc:1035] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:26:10.183573 32367 consensus_queue.cc:1035] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:26:10.186192 32386 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:26:10.186890 30440 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:26:10.186944 30571 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:26:10.189942 28462 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 876004fec43e428e930f3fbb873ccac8 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:10.193490 28475 catalog_manager.cc:5484] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8 reported cstate change: config changed from index 9 to 10, VOTER 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66) evicted. New cstate: current_term: 1 leader_uuid: "52c5becf6262415cab0408b3968ed4c0" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:26:10.197712 28735 tablet_service.cc:1514] Processing DeleteTablet for tablet 876004fec43e428e930f3fbb873ccac8 with delete_type TABLET_DATA_TOMBSTONED (TS 8786a1419d3f4bc79b88c54ce0675aa6 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:34018
I20240502 21:26:10.198801 32394 tablet_replica.cc:330] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6: stopping tablet replica
I20240502 21:26:10.199081 32394 raft_consensus.cc:2229] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:10.199748 32394 raft_consensus.cc:2258] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:10.201411 32394 ts_tablet_manager.cc:1875] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:10.206733 32394 ts_tablet_manager.cc:1888] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 21:26:10.206877 32394 log.cc:1196] T 876004fec43e428e930f3fbb873ccac8 P 8786a1419d3f4bc79b88c54ce0675aa6: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/wals/876004fec43e428e930f3fbb873ccac8
I20240502 21:26:10.207854 28461 catalog_manager.cc:4830] TS 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815): tablet 876004fec43e428e930f3fbb873ccac8 (table rebalance_test_table_2 [id=125451552e264fde833414d8cda8be18]) successfully deleted
W20240502 21:26:10.215093 28824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 21:26:10.300146 28824 consensus_peers.cc:485] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:26:10.322887 28961 consensus_peers.cc:485] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:26:10.323201 28687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 21:26:10.396459 29006 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:26:10.406322 30552 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:10.420464 28871 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:26:10.443835 30152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:10.459476 28735 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:26:10.462110 30420 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:10.502468 30285 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:10.591183 29029 consensus_queue.cc:237] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } }
I20240502 21:26:10.594252 28893 raft_consensus.cc:1261] T 5370969bf2f44ee0833fafa4285edf9f P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:26:10.594234 30440 raft_consensus.cc:1261] T 5370969bf2f44ee0833fafa4285edf9f P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:26:10.595125 32386 consensus_queue.cc:1035] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:26:10.595572 32367 consensus_queue.cc:1035] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:26:10.599313 32367 raft_consensus.cc:2932] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } } }
I20240502 21:26:10.600648 28893 raft_consensus.cc:2932] T 5370969bf2f44ee0833fafa4285edf9f P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } } }
I20240502 21:26:10.600739 30440 raft_consensus.cc:2932] T 5370969bf2f44ee0833fafa4285edf9f P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } } }
W20240502 21:26:10.601850 28963 consensus_peers.cc:485] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 -> Peer fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70:43969): Couldn't send request to peer fc5aa2dc1f7b4554906785d693e272d0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5370969bf2f44ee0833fafa4285edf9f. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:10.607193 28476 catalog_manager.cc:5484] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 reported cstate change: config changed from index 6 to 7, NON_VOTER fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) added. New cstate: current_term: 1 leader_uuid: "52c5becf6262415cab0408b3968ed4c0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:10.623785 28893 consensus_queue.cc:237] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: true } }
W20240502 21:26:10.625118 28824 consensus_peers.cc:485] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 21:26:10.627564 30440 raft_consensus.cc:1261] T 2c630e4bed554c8fa99ecfabed6519f1 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:26:10.627652 28755 raft_consensus.cc:1261] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 21:26:10.627771 28824 consensus_peers.cc:485] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 -> Peer a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69:41545): Couldn't send request to peer a1e2dba8e1a3403295cb394f7090f849. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c630e4bed554c8fa99ecfabed6519f1. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:10.628525 32450 consensus_queue.cc:1035] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:26:10.629101 32347 consensus_queue.cc:1035] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:26:10.634099 28755 raft_consensus.cc:2932] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: true } } }
I20240502 21:26:10.634351 32450 raft_consensus.cc:2932] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: true } } }
I20240502 21:26:10.640772 28467 catalog_manager.cc:5484] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6 reported cstate change: config changed from index 6 to 7, NON_VOTER a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) added. New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: true } } }
I20240502 21:26:10.649005 30440 raft_consensus.cc:2932] T 2c630e4bed554c8fa99ecfabed6519f1 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: true } } }
W20240502 21:26:10.786716 28961 consensus_peers.cc:485] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 21:26:11.086143 32473 ts_tablet_manager.cc:902] T 2c630e4bed554c8fa99ecfabed6519f1 P a1e2dba8e1a3403295cb394f7090f849: Initiating tablet copy from peer 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545)
I20240502 21:26:11.087776 32473 tablet_copy_client.cc:308] T 2c630e4bed554c8fa99ecfabed6519f1 P a1e2dba8e1a3403295cb394f7090f849: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:42545
I20240502 21:26:11.088738 28913 tablet_copy_service.cc:140] P 8733381e64da4e91a8a54e23b01e8967: Received BeginTabletCopySession request for tablet 2c630e4bed554c8fa99ecfabed6519f1 from peer a1e2dba8e1a3403295cb394f7090f849 ({username='slave'} at 127.27.196.69:38737)
I20240502 21:26:11.088915 28913 tablet_copy_service.cc:161] P 8733381e64da4e91a8a54e23b01e8967: Beginning new tablet copy session on tablet 2c630e4bed554c8fa99ecfabed6519f1 from peer a1e2dba8e1a3403295cb394f7090f849 at {username='slave'} at 127.27.196.69:38737: session id = a1e2dba8e1a3403295cb394f7090f849-2c630e4bed554c8fa99ecfabed6519f1
I20240502 21:26:11.091024 28913 tablet_copy_source_session.cc:205] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:11.092772 32473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c630e4bed554c8fa99ecfabed6519f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:11.097746 32473 tablet_copy_client.cc:783] T 2c630e4bed554c8fa99ecfabed6519f1 P a1e2dba8e1a3403295cb394f7090f849: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:11.098045 32473 tablet_copy_client.cc:647] T 2c630e4bed554c8fa99ecfabed6519f1 P a1e2dba8e1a3403295cb394f7090f849: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:11.100450 32477 tablet_copy_client.cc:1002] T 2c630e4bed554c8fa99ecfabed6519f1 P a1e2dba8e1a3403295cb394f7090f849: tablet copy: Injecting latency into file download: 1500
I20240502 21:26:11.171038 32488 ts_tablet_manager.cc:902] T 5370969bf2f44ee0833fafa4285edf9f P fc5aa2dc1f7b4554906785d693e272d0: Initiating tablet copy from peer 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795)
I20240502 21:26:11.172968 32488 tablet_copy_client.cc:308] T 5370969bf2f44ee0833fafa4285edf9f P fc5aa2dc1f7b4554906785d693e272d0: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.68:41795
I20240502 21:26:11.180224 29050 tablet_copy_service.cc:140] P 52c5becf6262415cab0408b3968ed4c0: Received BeginTabletCopySession request for tablet 5370969bf2f44ee0833fafa4285edf9f from peer fc5aa2dc1f7b4554906785d693e272d0 ({username='slave'} at 127.27.196.70:55299)
I20240502 21:26:11.180526 29050 tablet_copy_service.cc:161] P 52c5becf6262415cab0408b3968ed4c0: Beginning new tablet copy session on tablet 5370969bf2f44ee0833fafa4285edf9f from peer fc5aa2dc1f7b4554906785d693e272d0 at {username='slave'} at 127.27.196.70:55299: session id = fc5aa2dc1f7b4554906785d693e272d0-5370969bf2f44ee0833fafa4285edf9f
I20240502 21:26:11.182997 29050 tablet_copy_source_session.cc:205] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:11.185029 32488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5370969bf2f44ee0833fafa4285edf9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:11.191231 32488 tablet_copy_client.cc:783] T 5370969bf2f44ee0833fafa4285edf9f P fc5aa2dc1f7b4554906785d693e272d0: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:11.191900 32488 tablet_copy_client.cc:647] T 5370969bf2f44ee0833fafa4285edf9f P fc5aa2dc1f7b4554906785d693e272d0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:11.194211 32492 tablet_copy_client.cc:1002] T 5370969bf2f44ee0833fafa4285edf9f P fc5aa2dc1f7b4554906785d693e272d0: tablet copy: Injecting latency into file download: 1500
W20240502 21:26:11.868918 28687 consensus_peers.cc:485] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:26:12.155084 28824 consensus_peers.cc:485] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:26:12.504750 28687 consensus_peers.cc:485] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 21:26:12.601049 32473 tablet_copy_client.cc:523] T 2c630e4bed554c8fa99ecfabed6519f1 P a1e2dba8e1a3403295cb394f7090f849: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:12.604293 32473 tablet_bootstrap.cc:492] T 2c630e4bed554c8fa99ecfabed6519f1 P a1e2dba8e1a3403295cb394f7090f849: Bootstrap starting.
I20240502 21:26:12.640203 32473 tablet_bootstrap.cc:492] T 2c630e4bed554c8fa99ecfabed6519f1 P a1e2dba8e1a3403295cb394f7090f849: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:12.640682 32473 tablet_bootstrap.cc:492] T 2c630e4bed554c8fa99ecfabed6519f1 P a1e2dba8e1a3403295cb394f7090f849: Bootstrap complete.
I20240502 21:26:12.640974 32473 ts_tablet_manager.cc:1367] T 2c630e4bed554c8fa99ecfabed6519f1 P a1e2dba8e1a3403295cb394f7090f849: Time spent bootstrapping tablet: real 0.037s	user 0.031s	sys 0.004s
I20240502 21:26:12.641764 32473 raft_consensus.cc:348] T 2c630e4bed554c8fa99ecfabed6519f1 P a1e2dba8e1a3403295cb394f7090f849 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: true } }
I20240502 21:26:12.642014 32473 raft_consensus.cc:729] T 2c630e4bed554c8fa99ecfabed6519f1 P a1e2dba8e1a3403295cb394f7090f849 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a1e2dba8e1a3403295cb394f7090f849, State: Initialized, Role: LEARNER
I20240502 21:26:12.642277 32473 consensus_queue.cc:260] T 2c630e4bed554c8fa99ecfabed6519f1 P a1e2dba8e1a3403295cb394f7090f849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: true } }
I20240502 21:26:12.643292 32473 ts_tablet_manager.cc:1398] T 2c630e4bed554c8fa99ecfabed6519f1 P a1e2dba8e1a3403295cb394f7090f849: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.001s
I20240502 21:26:12.644232 28913 tablet_copy_service.cc:342] P 8733381e64da4e91a8a54e23b01e8967: Request end of tablet copy session a1e2dba8e1a3403295cb394f7090f849-2c630e4bed554c8fa99ecfabed6519f1 received from {username='slave'} at 127.27.196.69:38737
I20240502 21:26:12.644440 28913 tablet_copy_service.cc:433] P 8733381e64da4e91a8a54e23b01e8967: ending tablet copy session a1e2dba8e1a3403295cb394f7090f849-2c630e4bed554c8fa99ecfabed6519f1 on tablet 2c630e4bed554c8fa99ecfabed6519f1 with peer a1e2dba8e1a3403295cb394f7090f849
W20240502 21:26:12.656527 28961 consensus_peers.cc:485] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 21:26:12.694710 32488 tablet_copy_client.cc:523] T 5370969bf2f44ee0833fafa4285edf9f P fc5aa2dc1f7b4554906785d693e272d0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:12.698174 32488 tablet_bootstrap.cc:492] T 5370969bf2f44ee0833fafa4285edf9f P fc5aa2dc1f7b4554906785d693e272d0: Bootstrap starting.
I20240502 21:26:12.738868 32488 tablet_bootstrap.cc:492] T 5370969bf2f44ee0833fafa4285edf9f P fc5aa2dc1f7b4554906785d693e272d0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:12.739475 32488 tablet_bootstrap.cc:492] T 5370969bf2f44ee0833fafa4285edf9f P fc5aa2dc1f7b4554906785d693e272d0: Bootstrap complete.
I20240502 21:26:12.739791 32488 ts_tablet_manager.cc:1367] T 5370969bf2f44ee0833fafa4285edf9f P fc5aa2dc1f7b4554906785d693e272d0: Time spent bootstrapping tablet: real 0.042s	user 0.034s	sys 0.009s
I20240502 21:26:12.740705 32488 raft_consensus.cc:348] T 5370969bf2f44ee0833fafa4285edf9f P fc5aa2dc1f7b4554906785d693e272d0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } }
I20240502 21:26:12.740971 32488 raft_consensus.cc:729] T 5370969bf2f44ee0833fafa4285edf9f P fc5aa2dc1f7b4554906785d693e272d0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fc5aa2dc1f7b4554906785d693e272d0, State: Initialized, Role: LEARNER
I20240502 21:26:12.741191 32488 consensus_queue.cc:260] T 5370969bf2f44ee0833fafa4285edf9f P fc5aa2dc1f7b4554906785d693e272d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } }
I20240502 21:26:12.742084 32488 ts_tablet_manager.cc:1398] T 5370969bf2f44ee0833fafa4285edf9f P fc5aa2dc1f7b4554906785d693e272d0: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240502 21:26:12.743842 29050 tablet_copy_service.cc:342] P 52c5becf6262415cab0408b3968ed4c0: Request end of tablet copy session fc5aa2dc1f7b4554906785d693e272d0-5370969bf2f44ee0833fafa4285edf9f received from {username='slave'} at 127.27.196.70:55299
I20240502 21:26:12.744001 29050 tablet_copy_service.cc:433] P 52c5becf6262415cab0408b3968ed4c0: ending tablet copy session fc5aa2dc1f7b4554906785d693e272d0-5370969bf2f44ee0833fafa4285edf9f on tablet 5370969bf2f44ee0833fafa4285edf9f with peer fc5aa2dc1f7b4554906785d693e272d0
I20240502 21:26:12.799299 30172 raft_consensus.cc:1203] T 2c630e4bed554c8fa99ecfabed6519f1 P a1e2dba8e1a3403295cb394f7090f849 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 21:26:12.803004 30304 raft_consensus.cc:1203] T 5370969bf2f44ee0833fafa4285edf9f P fc5aa2dc1f7b4554906785d693e272d0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
W20240502 21:26:12.929252 28824 consensus_peers.cc:485] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:26:12.993285 28824 consensus_peers.cc:485] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:26:13.169659 28461 catalog_manager.cc:4573] TS 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3665b7b9141b4fe8965eeecea35e9c0b: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
I20240502 21:26:13.237344 32603 raft_consensus.cc:1050] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0: attempting to promote NON_VOTER fc5aa2dc1f7b4554906785d693e272d0 to VOTER
I20240502 21:26:13.238144 32603 consensus_queue.cc:237] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } }
I20240502 21:26:13.241071 28893 raft_consensus.cc:1261] T 5370969bf2f44ee0833fafa4285edf9f P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:13.241111 30304 raft_consensus.cc:1261] T 5370969bf2f44ee0833fafa4285edf9f P fc5aa2dc1f7b4554906785d693e272d0 [term 1 LEARNER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:13.241083 30440 raft_consensus.cc:1261] T 5370969bf2f44ee0833fafa4285edf9f P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:13.241739 32603 consensus_queue.cc:1035] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:13.242194 32605 consensus_queue.cc:1035] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:13.242604 32556 consensus_queue.cc:1035] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:13.246353 32605 raft_consensus.cc:2932] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:26:13.247749 30304 raft_consensus.cc:2932] T 5370969bf2f44ee0833fafa4285edf9f P fc5aa2dc1f7b4554906785d693e272d0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:26:13.250356 28893 raft_consensus.cc:2932] T 5370969bf2f44ee0833fafa4285edf9f P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:26:13.254417 28467 catalog_manager.cc:5484] T 5370969bf2f44ee0833fafa4285edf9f P fc5aa2dc1f7b4554906785d693e272d0 reported cstate change: config changed from index 7 to 8, fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "52c5becf6262415cab0408b3968ed4c0" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:26:13.256592 30440 raft_consensus.cc:2932] T 5370969bf2f44ee0833fafa4285edf9f P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:26:13.264139 29029 consensus_queue.cc:237] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } }
I20240502 21:26:13.267597 30304 raft_consensus.cc:1261] T 5370969bf2f44ee0833fafa4285edf9f P fc5aa2dc1f7b4554906785d693e272d0 [term 1 FOLLOWER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:26:13.268709 32603 consensus_queue.cc:1035] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:26:13.272660 30440 raft_consensus.cc:1261] T 5370969bf2f44ee0833fafa4285edf9f P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:26:13.272469 32556 raft_consensus.cc:2932] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:26:13.272974 30304 raft_consensus.cc:2932] T 5370969bf2f44ee0833fafa4285edf9f P fc5aa2dc1f7b4554906785d693e272d0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:26:13.273437 32605 consensus_queue.cc:1035] T 5370969bf2f44ee0833fafa4285edf9f P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:26:13.275342 30440 raft_consensus.cc:2932] T 5370969bf2f44ee0833fafa4285edf9f P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:26:13.278293 28462 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5370969bf2f44ee0833fafa4285edf9f with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:13.279844 28476 catalog_manager.cc:5484] T 5370969bf2f44ee0833fafa4285edf9f P fc5aa2dc1f7b4554906785d693e272d0 reported cstate change: config changed from index 8 to 9, VOTER 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67) evicted. New cstate: current_term: 1 leader_uuid: "52c5becf6262415cab0408b3968ed4c0" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:26:13.289412 28871 tablet_service.cc:1514] Processing DeleteTablet for tablet 5370969bf2f44ee0833fafa4285edf9f with delete_type TABLET_DATA_TOMBSTONED (TS 8733381e64da4e91a8a54e23b01e8967 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:53992
I20240502 21:26:13.293380 32614 raft_consensus.cc:1050] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967: attempting to promote NON_VOTER a1e2dba8e1a3403295cb394f7090f849 to VOTER
I20240502 21:26:13.295903 32617 tablet_replica.cc:330] T 5370969bf2f44ee0833fafa4285edf9f P 8733381e64da4e91a8a54e23b01e8967: stopping tablet replica
I20240502 21:26:13.296190 32617 raft_consensus.cc:2229] T 5370969bf2f44ee0833fafa4285edf9f P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:13.296653 32617 raft_consensus.cc:2258] T 5370969bf2f44ee0833fafa4285edf9f P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:13.296478 32614 consensus_queue.cc:237] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } }
I20240502 21:26:13.299221 32617 ts_tablet_manager.cc:1875] T 5370969bf2f44ee0833fafa4285edf9f P 8733381e64da4e91a8a54e23b01e8967: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:13.299607 30172 raft_consensus.cc:1261] T 2c630e4bed554c8fa99ecfabed6519f1 P a1e2dba8e1a3403295cb394f7090f849 [term 1 LEARNER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:13.299962 28755 raft_consensus.cc:1261] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:13.299963 30440 raft_consensus.cc:1261] T 2c630e4bed554c8fa99ecfabed6519f1 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:13.300312 32615 consensus_queue.cc:1035] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:13.300781 32542 consensus_queue.cc:1035] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:13.301131 32614 consensus_queue.cc:1035] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:13.306248 32615 raft_consensus.cc:2932] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
I20240502 21:26:13.307040 30172 raft_consensus.cc:2932] T 2c630e4bed554c8fa99ecfabed6519f1 P a1e2dba8e1a3403295cb394f7090f849 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
I20240502 21:26:13.307719 30440 raft_consensus.cc:2932] T 2c630e4bed554c8fa99ecfabed6519f1 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
I20240502 21:26:13.310942 28755 raft_consensus.cc:2932] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
I20240502 21:26:13.312196 28471 catalog_manager.cc:5484] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 reported cstate change: config changed from index 7 to 8, a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:13.318082 32617 ts_tablet_manager.cc:1888] T 5370969bf2f44ee0833fafa4285edf9f P 8733381e64da4e91a8a54e23b01e8967: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:26:13.318226 32617 log.cc:1196] T 5370969bf2f44ee0833fafa4285edf9f P 8733381e64da4e91a8a54e23b01e8967: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/wals/5370969bf2f44ee0833fafa4285edf9f
I20240502 21:26:13.318527 28893 consensus_queue.cc:237] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } }
I20240502 21:26:13.319161 28460 catalog_manager.cc:4830] TS 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545): tablet 5370969bf2f44ee0833fafa4285edf9f (table rebalance_test_table_1 [id=a4cb53742e604e37ae1a6151409c162b]) successfully deleted
I20240502 21:26:13.323880 30440 raft_consensus.cc:1261] T 2c630e4bed554c8fa99ecfabed6519f1 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:26:13.324563 32615 consensus_queue.cc:1035] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:26:13.325649 30171 raft_consensus.cc:1261] T 2c630e4bed554c8fa99ecfabed6519f1 P a1e2dba8e1a3403295cb394f7090f849 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:26:13.327991 32615 consensus_queue.cc:1035] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:26:13.328733 32542 raft_consensus.cc:2932] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
I20240502 21:26:13.329723 30440 raft_consensus.cc:2932] T 2c630e4bed554c8fa99ecfabed6519f1 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
I20240502 21:26:13.333040 28460 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2c630e4bed554c8fa99ecfabed6519f1 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:13.332937 30171 raft_consensus.cc:2932] T 2c630e4bed554c8fa99ecfabed6519f1 P a1e2dba8e1a3403295cb394f7090f849 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } }
I20240502 21:26:13.337399 28467 catalog_manager.cc:5484] T 2c630e4bed554c8fa99ecfabed6519f1 P 8733381e64da4e91a8a54e23b01e8967 reported cstate change: config changed from index 8 to 9, VOTER 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66) evicted. New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:13.342151 28735 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c630e4bed554c8fa99ecfabed6519f1 with delete_type TABLET_DATA_TOMBSTONED (TS 8786a1419d3f4bc79b88c54ce0675aa6 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:34018
I20240502 21:26:13.343851 32623 tablet_replica.cc:330] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6: stopping tablet replica
I20240502 21:26:13.344120 32623 raft_consensus.cc:2229] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:13.344455 32623 raft_consensus.cc:2258] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:13.345623 32623 ts_tablet_manager.cc:1875] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:13.350330 32623 ts_tablet_manager.cc:1888] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:26:13.350471 32623 log.cc:1196] T 2c630e4bed554c8fa99ecfabed6519f1 P 8786a1419d3f4bc79b88c54ce0675aa6: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/wals/2c630e4bed554c8fa99ecfabed6519f1
I20240502 21:26:13.351397 28461 catalog_manager.cc:4830] TS 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815): tablet 2c630e4bed554c8fa99ecfabed6519f1 (table rebalance_test_table_3 [id=e6d6bbf7aaec4d548ddf673c06866772]) successfully deleted
W20240502 21:26:13.427822 28961 consensus_peers.cc:485] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 21:26:13.468238 28893 consensus_queue.cc:237] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: NON_VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: true } }
I20240502 21:26:13.470796 30304 raft_consensus.cc:1261] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240502 21:26:13.471361 28825 consensus_peers.cc:485] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 -> Peer a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71:36827): Couldn't send request to peer a9ca254f89984b3f9ed62020bd6a4d02. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4c6f9c73a2d4b01a2c027504e45b238. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:13.471405 30571 raft_consensus.cc:1261] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:26:13.471490 32614 consensus_queue.cc:1035] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:26:13.472016 32615 consensus_queue.cc:1035] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:26:13.474483 32614 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: NON_VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: true } } }
I20240502 21:26:13.475667 30304 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: NON_VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: true } } }
I20240502 21:26:13.475855 30571 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: NON_VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: true } } }
I20240502 21:26:13.482004 28471 catalog_manager.cc:5484] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 reported cstate change: config changed from index 10 to 11, NON_VOTER a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) added. New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: NON_VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:13.510825 29029 consensus_queue.cc:237] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } }
I20240502 21:26:13.513687 30440 raft_consensus.cc:1261] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:26:13.513758 30571 raft_consensus.cc:1261] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 FOLLOWER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240502 21:26:13.513864 28963 consensus_peers.cc:485] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 -> Peer fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70:43969): Couldn't send request to peer fc5aa2dc1f7b4554906785d693e272d0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 876004fec43e428e930f3fbb873ccac8. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:13.514371 32556 consensus_queue.cc:1035] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:26:13.514773 32603 consensus_queue.cc:1035] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:26:13.518394 32556 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } } }
I20240502 21:26:13.519032 30571 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } } }
I20240502 21:26:13.520294 30440 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } } }
I20240502 21:26:13.524219 28475 catalog_manager.cc:5484] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 reported cstate change: config changed from index 10 to 11, NON_VOTER fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) added. New cstate: current_term: 1 leader_uuid: "52c5becf6262415cab0408b3968ed4c0" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:13.904057 32671 ts_tablet_manager.cc:902] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0: Initiating tablet copy from peer 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795)
I20240502 21:26:13.905750 32671 tablet_copy_client.cc:308] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.68:41795
I20240502 21:26:13.906806 29050 tablet_copy_service.cc:140] P 52c5becf6262415cab0408b3968ed4c0: Received BeginTabletCopySession request for tablet 876004fec43e428e930f3fbb873ccac8 from peer fc5aa2dc1f7b4554906785d693e272d0 ({username='slave'} at 127.27.196.70:55299)
I20240502 21:26:13.906997 29050 tablet_copy_service.cc:161] P 52c5becf6262415cab0408b3968ed4c0: Beginning new tablet copy session on tablet 876004fec43e428e930f3fbb873ccac8 from peer fc5aa2dc1f7b4554906785d693e272d0 at {username='slave'} at 127.27.196.70:55299: session id = fc5aa2dc1f7b4554906785d693e272d0-876004fec43e428e930f3fbb873ccac8
I20240502 21:26:13.909370 29050 tablet_copy_source_session.cc:205] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:13.910882 32671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 876004fec43e428e930f3fbb873ccac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:13.915573 32671 tablet_copy_client.cc:783] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:13.915858 32671 tablet_copy_client.cc:647] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:13.917737 32675 tablet_copy_client.cc:1002] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0: tablet copy: Injecting latency into file download: 1500
I20240502 21:26:13.986353 32681 ts_tablet_manager.cc:902] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02: Initiating tablet copy from peer 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545)
I20240502 21:26:13.988150 32681 tablet_copy_client.cc:308] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:42545
I20240502 21:26:13.989251 28913 tablet_copy_service.cc:140] P 8733381e64da4e91a8a54e23b01e8967: Received BeginTabletCopySession request for tablet a4c6f9c73a2d4b01a2c027504e45b238 from peer a9ca254f89984b3f9ed62020bd6a4d02 ({username='slave'} at 127.27.196.71:41703)
I20240502 21:26:13.989440 28913 tablet_copy_service.cc:161] P 8733381e64da4e91a8a54e23b01e8967: Beginning new tablet copy session on tablet a4c6f9c73a2d4b01a2c027504e45b238 from peer a9ca254f89984b3f9ed62020bd6a4d02 at {username='slave'} at 127.27.196.71:41703: session id = a9ca254f89984b3f9ed62020bd6a4d02-a4c6f9c73a2d4b01a2c027504e45b238
I20240502 21:26:13.992034 28913 tablet_copy_source_session.cc:205] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:13.993680 32681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c6f9c73a2d4b01a2c027504e45b238. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:14.000082 32681 tablet_copy_client.cc:783] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:14.000337 32681 tablet_copy_client.cc:647] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:14.002128 32685 tablet_copy_client.cc:1002] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02: tablet copy: Injecting latency into file download: 1500
I20240502 21:26:14.233887 31892 consensus_queue.cc:579] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Leader has been unable to successfully communicate with peer 04e36797791842e489aee30de261c4c7 for more than 30 seconds (30.267s)
W20240502 21:26:14.236244 28687 consensus_peers.cc:485] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 21:26:14.332589 32542 consensus_queue.cc:579] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Leader has been unable to successfully communicate with peer 04e36797791842e489aee30de261c4c7 for more than 30 seconds (30.031s)
I20240502 21:26:14.471745 32703 consensus_queue.cc:579] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Leader has been unable to successfully communicate with peer 04e36797791842e489aee30de261c4c7 for more than 30 seconds (30.133s)
I20240502 21:26:14.479028 32603 consensus_queue.cc:579] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Leader has been unable to successfully communicate with peer 04e36797791842e489aee30de261c4c7 for more than 30 seconds (30.187s)
I20240502 21:26:14.706924 32705 consensus_queue.cc:579] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Leader has been unable to successfully communicate with peer 04e36797791842e489aee30de261c4c7 for more than 30 seconds (30.115s)
W20240502 21:26:14.709633 28824 consensus_peers.cc:485] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 21:26:14.824869 32705 consensus_queue.cc:579] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Leader has been unable to successfully communicate with peer 04e36797791842e489aee30de261c4c7 for more than 30 seconds (30.233s)
W20240502 21:26:14.886437 28687 consensus_peers.cc:485] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:26:14.939086 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:26:15.084565 28961 consensus_peers.cc:485] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:26:15.265118 28824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 21:26:15.266487 28824 consensus_peers.cc:485] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 21:26:15.418339 32671 tablet_copy_client.cc:523] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:15.421484 32671 tablet_bootstrap.cc:492] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0: Bootstrap starting.
W20240502 21:26:15.436920 28824 consensus_peers.cc:485] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:26:15.454448 28687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 21:26:15.459669 32671 tablet_bootstrap.cc:492] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:15.460124 32671 tablet_bootstrap.cc:492] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0: Bootstrap complete.
I20240502 21:26:15.460376 32671 ts_tablet_manager.cc:1367] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0: Time spent bootstrapping tablet: real 0.039s	user 0.040s	sys 0.000s
I20240502 21:26:15.461079 32671 raft_consensus.cc:348] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } }
I20240502 21:26:15.461293 32671 raft_consensus.cc:729] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fc5aa2dc1f7b4554906785d693e272d0, State: Initialized, Role: LEARNER
I20240502 21:26:15.461458 32671 consensus_queue.cc:260] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: NON_VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: true } }
I20240502 21:26:15.462343 32671 ts_tablet_manager.cc:1398] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 21:26:15.463620 29050 tablet_copy_service.cc:342] P 52c5becf6262415cab0408b3968ed4c0: Request end of tablet copy session fc5aa2dc1f7b4554906785d693e272d0-876004fec43e428e930f3fbb873ccac8 received from {username='slave'} at 127.27.196.70:55299
I20240502 21:26:15.463762 29050 tablet_copy_service.cc:433] P 52c5becf6262415cab0408b3968ed4c0: ending tablet copy session fc5aa2dc1f7b4554906785d693e272d0-876004fec43e428e930f3fbb873ccac8 on tablet 876004fec43e428e930f3fbb873ccac8 with peer fc5aa2dc1f7b4554906785d693e272d0
I20240502 21:26:15.502830 32681 tablet_copy_client.cc:523] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:15.506561 32681 tablet_bootstrap.cc:492] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02: Bootstrap starting.
I20240502 21:26:15.530938 30304 raft_consensus.cc:1203] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11]   Dedup: 1.11->[]
I20240502 21:26:15.546231 32681 tablet_bootstrap.cc:492] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:15.546813 32681 tablet_bootstrap.cc:492] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02: Bootstrap complete.
I20240502 21:26:15.547142 32681 ts_tablet_manager.cc:1367] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02: Time spent bootstrapping tablet: real 0.041s	user 0.030s	sys 0.010s
I20240502 21:26:15.547878 32681 raft_consensus.cc:348] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: NON_VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: true } }
I20240502 21:26:15.548111 32681 raft_consensus.cc:729] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a9ca254f89984b3f9ed62020bd6a4d02, State: Initialized, Role: LEARNER
I20240502 21:26:15.548310 32681 consensus_queue.cc:260] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: NON_VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: true } }
I20240502 21:26:15.549248 32681 ts_tablet_manager.cc:1398] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 21:26:15.550354 28913 tablet_copy_service.cc:342] P 8733381e64da4e91a8a54e23b01e8967: Request end of tablet copy session a9ca254f89984b3f9ed62020bd6a4d02-a4c6f9c73a2d4b01a2c027504e45b238 received from {username='slave'} at 127.27.196.71:41703
I20240502 21:26:15.550489 28913 tablet_copy_service.cc:433] P 8733381e64da4e91a8a54e23b01e8967: ending tablet copy session a9ca254f89984b3f9ed62020bd6a4d02-a4c6f9c73a2d4b01a2c027504e45b238 on tablet a4c6f9c73a2d4b01a2c027504e45b238 with peer a9ca254f89984b3f9ed62020bd6a4d02
I20240502 21:26:15.671409 30440 raft_consensus.cc:1203] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11]   Dedup: 1.11->[]
I20240502 21:26:15.916846 32603 consensus_queue.cc:579] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Leader has been unable to successfully communicate with peer 04e36797791842e489aee30de261c4c7 for more than 30 seconds (30.198s)
W20240502 21:26:15.919275 28961 consensus_peers.cc:485] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 21:26:15.921849   313 raft_consensus.cc:1050] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0: attempting to promote NON_VOTER fc5aa2dc1f7b4554906785d693e272d0 to VOTER
I20240502 21:26:15.922765   313 consensus_queue.cc:237] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } }
I20240502 21:26:15.925858 30571 raft_consensus.cc:1261] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 FOLLOWER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:26:15.926136 30440 raft_consensus.cc:1261] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:26:15.926990 30304 raft_consensus.cc:1261] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 LEARNER]: Refusing update from remote peer 52c5becf6262415cab0408b3968ed4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:26:15.927479   314 consensus_queue.cc:1035] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:26:15.927984 32603 consensus_queue.cc:1035] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:26:15.928676   313 consensus_queue.cc:1035] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:26:15.932751   313 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:26:15.933806 30440 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:26:15.934334 30304 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:26:15.940795 30571 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:26:15.941037 28475 catalog_manager.cc:5484] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 reported cstate change: config changed from index 11 to 12, fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "52c5becf6262415cab0408b3968ed4c0" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:16.051507 29029 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "876004fec43e428e930f3fbb873ccac8"
dest_uuid: "52c5becf6262415cab0408b3968ed4c0"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:36412
I20240502 21:26:16.051831 29029 raft_consensus.cc:595] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:16.165057   342 raft_consensus.cc:1050] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967: attempting to promote NON_VOTER a9ca254f89984b3f9ed62020bd6a4d02 to VOTER
I20240502 21:26:16.166162   342 consensus_queue.cc:237] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } }
I20240502 21:26:16.169930 30304 raft_consensus.cc:1261] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:26:16.169962 30440 raft_consensus.cc:1261] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 LEARNER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:26:16.170774   344 consensus_queue.cc:1035] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:26:16.171263 30571 raft_consensus.cc:1261] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:26:16.171373   342 consensus_queue.cc:1035] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:26:16.171936 32705 consensus_queue.cc:1035] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:26:16.176560   342 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:26:16.177803 30304 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:26:16.178015 30571 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:26:16.181164 30440 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:26:16.181841 28471 catalog_manager.cc:5484] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 reported cstate change: config changed from index 11 to 12, a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:16.288081 29029 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "876004fec43e428e930f3fbb873ccac8"
dest_uuid: "52c5becf6262415cab0408b3968ed4c0"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:36430
I20240502 21:26:16.288408 29029 raft_consensus.cc:595] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:16.309386 28893 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a4c6f9c73a2d4b01a2c027504e45b238"
dest_uuid: "8733381e64da4e91a8a54e23b01e8967"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:45948
I20240502 21:26:16.309731 28893 raft_consensus.cc:595] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:16.322957 32603 raft_consensus.cc:979] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0: : Instructing follower fc5aa2dc1f7b4554906785d693e272d0 to start an election
I20240502 21:26:16.323225 32603 raft_consensus.cc:1067] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 1 LEADER]: Signalling peer fc5aa2dc1f7b4554906785d693e272d0 to start an election
I20240502 21:26:16.324527 30304 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "876004fec43e428e930f3fbb873ccac8"
dest_uuid: "fc5aa2dc1f7b4554906785d693e272d0"
 from {username='slave'} at 127.27.196.68:54895
I20240502 21:26:16.325026 30304 raft_consensus.cc:482] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:16.325253 30304 raft_consensus.cc:3037] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:16.328321 30304 raft_consensus.cc:504] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } }
I20240502 21:26:16.330389 30304 leader_election.cc:288] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0 [CANDIDATE]: Term 2 election: Requested vote from peers 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795), a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71:36827), 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72:34997)
I20240502 21:26:16.331408 29029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876004fec43e428e930f3fbb873ccac8" candidate_uuid: "fc5aa2dc1f7b4554906785d693e272d0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "52c5becf6262415cab0408b3968ed4c0"
I20240502 21:26:16.331838 29029 raft_consensus.cc:3032] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:26:16.331974 29029 raft_consensus.cc:729] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 52c5becf6262415cab0408b3968ed4c0, State: Running, Role: LEADER
I20240502 21:26:16.332538 29029 consensus_queue.cc:260] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } }
I20240502 21:26:16.333276 29029 raft_consensus.cc:3037] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:16.337949 29029 raft_consensus.cc:2454] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc5aa2dc1f7b4554906785d693e272d0 in term 2.
I20240502 21:26:16.350731 30440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876004fec43e428e930f3fbb873ccac8" candidate_uuid: "fc5aa2dc1f7b4554906785d693e272d0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "a9ca254f89984b3f9ed62020bd6a4d02"
I20240502 21:26:16.351337 30440 raft_consensus.cc:3037] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:16.353708 30440 raft_consensus.cc:2454] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc5aa2dc1f7b4554906785d693e272d0 in term 2.
I20240502 21:26:16.354444 30240 leader_election.cc:302] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 52c5becf6262415cab0408b3968ed4c0, a9ca254f89984b3f9ed62020bd6a4d02, fc5aa2dc1f7b4554906785d693e272d0; no voters: 
I20240502 21:26:16.354947 30571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876004fec43e428e930f3fbb873ccac8" candidate_uuid: "fc5aa2dc1f7b4554906785d693e272d0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8"
I20240502 21:26:16.355332 30571 raft_consensus.cc:3037] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:16.357841 30571 raft_consensus.cc:2454] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc5aa2dc1f7b4554906785d693e272d0 in term 2.
I20240502 21:26:16.358829 32758 raft_consensus.cc:2781] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:26:16.359449 32758 raft_consensus.cc:686] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0 [term 2 LEADER]: Becoming Leader. State: Replica: fc5aa2dc1f7b4554906785d693e272d0, State: Running, Role: LEADER
I20240502 21:26:16.360065 32758 consensus_queue.cc:237] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } }
I20240502 21:26:16.364416 28471 catalog_manager.cc:5484] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0 reported cstate change: term changed from 1 to 2, leader changed from 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68) to fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70). New cstate: current_term: 2 leader_uuid: "fc5aa2dc1f7b4554906785d693e272d0" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 21:26:16.494784 28461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240502 21:26:16.496315 28461 catalog_manager.cc:4573] TS 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 876004fec43e428e930f3fbb873ccac8: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
I20240502 21:26:16.550258 28893 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a4c6f9c73a2d4b01a2c027504e45b238"
dest_uuid: "8733381e64da4e91a8a54e23b01e8967"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:45964
I20240502 21:26:16.550568 28893 raft_consensus.cc:595] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:16.671288   341 raft_consensus.cc:979] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967: : Instructing follower a9ca254f89984b3f9ed62020bd6a4d02 to start an election
I20240502 21:26:16.671597 32705 raft_consensus.cc:1067] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Signalling peer a9ca254f89984b3f9ed62020bd6a4d02 to start an election
I20240502 21:26:16.672716 30440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a4c6f9c73a2d4b01a2c027504e45b238"
dest_uuid: "a9ca254f89984b3f9ed62020bd6a4d02"
 from {username='slave'} at 127.27.196.67:47653
I20240502 21:26:16.673065 30440 raft_consensus.cc:482] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:16.673225 30440 raft_consensus.cc:3037] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:16.675889 30440 raft_consensus.cc:504] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } }
I20240502 21:26:16.677898 30440 leader_election.cc:288] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02 [CANDIDATE]: Term 2 election: Requested vote from peers 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545), 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72:34997), fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70:43969)
I20240502 21:26:16.678390 28893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c6f9c73a2d4b01a2c027504e45b238" candidate_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "8733381e64da4e91a8a54e23b01e8967"
I20240502 21:26:16.678840 28893 raft_consensus.cc:3032] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:26:16.678985 28893 raft_consensus.cc:729] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8733381e64da4e91a8a54e23b01e8967, State: Running, Role: LEADER
I20240502 21:26:16.679481 28893 consensus_queue.cc:260] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } }
I20240502 21:26:16.680698 28893 raft_consensus.cc:3037] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:16.683923 28893 raft_consensus.cc:2454] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9ca254f89984b3f9ed62020bd6a4d02 in term 2.
I20240502 21:26:16.689203 30571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c6f9c73a2d4b01a2c027504e45b238" candidate_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8"
I20240502 21:26:16.689481 30571 raft_consensus.cc:3037] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:16.689687 30304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c6f9c73a2d4b01a2c027504e45b238" candidate_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "fc5aa2dc1f7b4554906785d693e272d0"
I20240502 21:26:16.689946 30304 raft_consensus.cc:3037] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:16.691653 30571 raft_consensus.cc:2454] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9ca254f89984b3f9ed62020bd6a4d02 in term 2.
I20240502 21:26:16.692339 30304 raft_consensus.cc:2454] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9ca254f89984b3f9ed62020bd6a4d02 in term 2.
I20240502 21:26:16.692342 30373 leader_election.cc:302] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4e8bd1e7c1f848b0bdd2570de97506d8, 8733381e64da4e91a8a54e23b01e8967, a9ca254f89984b3f9ed62020bd6a4d02; no voters: 
I20240502 21:26:16.693363   394 raft_consensus.cc:2781] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:26:16.693876   394 raft_consensus.cc:686] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02 [term 2 LEADER]: Becoming Leader. State: Replica: a9ca254f89984b3f9ed62020bd6a4d02, State: Running, Role: LEADER
I20240502 21:26:16.694387   394 consensus_queue.cc:237] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } }
I20240502 21:26:16.698431 28471 catalog_manager.cc:5484] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02 reported cstate change: term changed from 1 to 2, leader changed from 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67) to a9ca254f89984b3f9ed62020bd6a4d02 (127.27.196.71). New cstate: current_term: 2 leader_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:16.757880 30440 raft_consensus.cc:1261] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02 [term 2 FOLLOWER]: Refusing update from remote peer fc5aa2dc1f7b4554906785d693e272d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 21:26:16.759078 32758 consensus_queue.cc:1035] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240502 21:26:16.807648 28687 consensus_peers.cc:485] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 21:26:16.882118 30571 raft_consensus.cc:1261] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 2 FOLLOWER]: Refusing update from remote peer fc5aa2dc1f7b4554906785d693e272d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 21:26:16.883085 32758 consensus_queue.cc:1035] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:26:16.889181 29029 raft_consensus.cc:1261] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 2 FOLLOWER]: Refusing update from remote peer fc5aa2dc1f7b4554906785d693e272d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 21:26:16.890210 32758 consensus_queue.cc:1035] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:26:16.910936 30304 consensus_queue.cc:237] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } }
I20240502 21:26:16.913522 30440 raft_consensus.cc:1261] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02 [term 2 FOLLOWER]: Refusing update from remote peer fc5aa2dc1f7b4554906785d693e272d0: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 21:26:16.913547 30571 raft_consensus.cc:1261] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 2 FOLLOWER]: Refusing update from remote peer fc5aa2dc1f7b4554906785d693e272d0: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 21:26:16.914258   407 consensus_queue.cc:1035] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:26:16.914815   409 consensus_queue.cc:1035] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:26:16.918720 32758 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0 [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:26:16.919921 30571 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:26:16.923207 28463 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 876004fec43e428e930f3fbb873ccac8 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:16.925412 30440 raft_consensus.cc:2932] T 876004fec43e428e930f3fbb873ccac8 P a9ca254f89984b3f9ed62020bd6a4d02 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } }
I20240502 21:26:16.925977 28467 catalog_manager.cc:5484] T 876004fec43e428e930f3fbb873ccac8 P fc5aa2dc1f7b4554906785d693e272d0 reported cstate change: config changed from index 12 to 14, VOTER 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68) evicted. New cstate: current_term: 2 leader_uuid: "fc5aa2dc1f7b4554906785d693e272d0" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:16.931294 29006 tablet_service.cc:1514] Processing DeleteTablet for tablet 876004fec43e428e930f3fbb873ccac8 with delete_type TABLET_DATA_TOMBSTONED (TS 52c5becf6262415cab0408b3968ed4c0 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:44172
I20240502 21:26:16.932154   413 tablet_replica.cc:330] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0: stopping tablet replica
I20240502 21:26:16.932371   413 raft_consensus.cc:2229] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:16.932685   413 raft_consensus.cc:2258] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:16.933571   413 ts_tablet_manager.cc:1875] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:16.940351   413 ts_tablet_manager.cc:1888] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20240502 21:26:16.940521   413 log.cc:1196] T 876004fec43e428e930f3fbb873ccac8 P 52c5becf6262415cab0408b3968ed4c0: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/wals/876004fec43e428e930f3fbb873ccac8
I20240502 21:26:16.941625 28462 catalog_manager.cc:4830] TS 52c5becf6262415cab0408b3968ed4c0 (127.27.196.68:41795): tablet 876004fec43e428e930f3fbb873ccac8 (table rebalance_test_table_2 [id=125451552e264fde833414d8cda8be18]) successfully deleted
W20240502 21:26:17.069475 28824 consensus_peers.cc:485] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 21:26:17.082789 28893 raft_consensus.cc:1261] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 2 FOLLOWER]: Refusing update from remote peer a9ca254f89984b3f9ed62020bd6a4d02: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 21:26:17.083981   394 consensus_queue.cc:1035] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:26:17.093797 30571 raft_consensus.cc:1261] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 2 FOLLOWER]: Refusing update from remote peer a9ca254f89984b3f9ed62020bd6a4d02: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 21:26:17.094532   394 consensus_queue.cc:1035] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:26:17.099046 30304 raft_consensus.cc:1261] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0 [term 2 FOLLOWER]: Refusing update from remote peer a9ca254f89984b3f9ed62020bd6a4d02: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 21:26:17.100914   430 consensus_queue.cc:1035] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240502 21:26:17.124512 28687 consensus_peers.cc:485] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 21:26:17.519470 30440 consensus_queue.cc:237] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } }
I20240502 21:26:17.522153 30304 raft_consensus.cc:1261] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0 [term 2 FOLLOWER]: Refusing update from remote peer a9ca254f89984b3f9ed62020bd6a4d02: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 21:26:17.522200 30571 raft_consensus.cc:1261] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 2 FOLLOWER]: Refusing update from remote peer a9ca254f89984b3f9ed62020bd6a4d02: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 21:26:17.523089   429 consensus_queue.cc:1035] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:26:17.523737   430 consensus_queue.cc:1035] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:26:17.526247   430 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P a9ca254f89984b3f9ed62020bd6a4d02 [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:26:17.527091 30304 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P fc5aa2dc1f7b4554906785d693e272d0 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:26:17.527315 30571 raft_consensus.cc:2932] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:26:17.531303 28462 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a4c6f9c73a2d4b01a2c027504e45b238 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:17.532299 28467 catalog_manager.cc:5484] T a4c6f9c73a2d4b01a2c027504e45b238 P 4e8bd1e7c1f848b0bdd2570de97506d8 reported cstate change: config changed from index 12 to 14, VOTER 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67) evicted. New cstate: current_term: 2 leader_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "a9ca254f89984b3f9ed62020bd6a4d02" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 36827 } attrs { promote: false } } }
I20240502 21:26:17.538342 28871 tablet_service.cc:1514] Processing DeleteTablet for tablet a4c6f9c73a2d4b01a2c027504e45b238 with delete_type TABLET_DATA_TOMBSTONED (TS 8733381e64da4e91a8a54e23b01e8967 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:53992
I20240502 21:26:17.539076   446 tablet_replica.cc:330] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967: stopping tablet replica
I20240502 21:26:17.539299   446 raft_consensus.cc:2229] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:17.539754   446 raft_consensus.cc:2258] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:17.541270   446 ts_tablet_manager.cc:1875] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:17.546753   446 ts_tablet_manager.cc:1888] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20240502 21:26:17.546912   446 log.cc:1196] T a4c6f9c73a2d4b01a2c027504e45b238 P 8733381e64da4e91a8a54e23b01e8967: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/wals/a4c6f9c73a2d4b01a2c027504e45b238
I20240502 21:26:17.548089 28460 catalog_manager.cc:4830] TS 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545): tablet a4c6f9c73a2d4b01a2c027504e45b238 (table rebalance_test_table_4 [id=9474d8b6526a4d809c3b7e67f807f596]) successfully deleted
W20240502 21:26:17.621201 28961 consensus_peers.cc:485] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:26:17.634109 28824 consensus_peers.cc:485] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 21:26:17.797784 30552 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:17.798815 28871 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:26:17.801348 29006 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 21:26:17.816470 28824 consensus_peers.cc:485] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 21:26:17.830845 30420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:17.848556 28735 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:26:17.857775 30152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:17.880324 30285 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:17.965744 28893 consensus_queue.cc:237] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } }
I20240502 21:26:17.969657 30171 raft_consensus.cc:1261] T c2cdd0cd6e344ca391d295f92c6f6658 P a1e2dba8e1a3403295cb394f7090f849 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:26:17.970351 28756 raft_consensus.cc:1261] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:26:17.970592 32705 consensus_queue.cc:1035] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 21:26:17.971071 28823 consensus_peers.cc:485] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 -> Peer 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72:34997): Couldn't send request to peer 4e8bd1e7c1f848b0bdd2570de97506d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2cdd0cd6e344ca391d295f92c6f6658. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:17.971073   499 consensus_queue.cc:1035] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:26:17.974583 32705 raft_consensus.cc:2932] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } } }
I20240502 21:26:17.975409 28756 raft_consensus.cc:2932] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } } }
I20240502 21:26:17.975454 30171 raft_consensus.cc:2932] T c2cdd0cd6e344ca391d295f92c6f6658 P a1e2dba8e1a3403295cb394f7090f849 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } } }
I20240502 21:26:17.980532 28472 catalog_manager.cc:5484] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 reported cstate change: config changed from index 6 to 7, NON_VOTER 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) added. New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } } }
I20240502 21:26:18.415257   516 ts_tablet_manager.cc:902] T c2cdd0cd6e344ca391d295f92c6f6658 P 4e8bd1e7c1f848b0bdd2570de97506d8: Initiating tablet copy from peer 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545)
I20240502 21:26:18.416582   516 tablet_copy_client.cc:308] T c2cdd0cd6e344ca391d295f92c6f6658 P 4e8bd1e7c1f848b0bdd2570de97506d8: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:42545
I20240502 21:26:18.417457 28913 tablet_copy_service.cc:140] P 8733381e64da4e91a8a54e23b01e8967: Received BeginTabletCopySession request for tablet c2cdd0cd6e344ca391d295f92c6f6658 from peer 4e8bd1e7c1f848b0bdd2570de97506d8 ({username='slave'} at 127.27.196.72:35103)
I20240502 21:26:18.417652 28913 tablet_copy_service.cc:161] P 8733381e64da4e91a8a54e23b01e8967: Beginning new tablet copy session on tablet c2cdd0cd6e344ca391d295f92c6f6658 from peer 4e8bd1e7c1f848b0bdd2570de97506d8 at {username='slave'} at 127.27.196.72:35103: session id = 4e8bd1e7c1f848b0bdd2570de97506d8-c2cdd0cd6e344ca391d295f92c6f6658
I20240502 21:26:18.420253 28913 tablet_copy_source_session.cc:205] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:18.422214   516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2cdd0cd6e344ca391d295f92c6f6658. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:18.429203   516 tablet_copy_client.cc:783] T c2cdd0cd6e344ca391d295f92c6f6658 P 4e8bd1e7c1f848b0bdd2570de97506d8: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:18.429544   516 tablet_copy_client.cc:647] T c2cdd0cd6e344ca391d295f92c6f6658 P 4e8bd1e7c1f848b0bdd2570de97506d8: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:18.432147   520 tablet_copy_client.cc:1002] T c2cdd0cd6e344ca391d295f92c6f6658 P 4e8bd1e7c1f848b0bdd2570de97506d8: tablet copy: Injecting latency into file download: 1500
W20240502 21:26:18.517010 28961 consensus_peers.cc:485] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:26:19.107748 28687 consensus_peers.cc:485] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:26:19.187674 28461 catalog_manager.cc:4573] TS 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a4c6f9c73a2d4b01a2c027504e45b238: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
W20240502 21:26:19.540223 28824 consensus_peers.cc:485] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:26:19.576004 28687 consensus_peers.cc:485] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 21:26:19.932750   516 tablet_copy_client.cc:523] T c2cdd0cd6e344ca391d295f92c6f6658 P 4e8bd1e7c1f848b0bdd2570de97506d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:19.935782   516 tablet_bootstrap.cc:492] T c2cdd0cd6e344ca391d295f92c6f6658 P 4e8bd1e7c1f848b0bdd2570de97506d8: Bootstrap starting.
W20240502 21:26:19.943593 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:26:19.944676 28961 consensus_peers.cc:485] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 21:26:19.975237   516 tablet_bootstrap.cc:492] T c2cdd0cd6e344ca391d295f92c6f6658 P 4e8bd1e7c1f848b0bdd2570de97506d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:19.975694   516 tablet_bootstrap.cc:492] T c2cdd0cd6e344ca391d295f92c6f6658 P 4e8bd1e7c1f848b0bdd2570de97506d8: Bootstrap complete.
I20240502 21:26:19.975971   516 ts_tablet_manager.cc:1367] T c2cdd0cd6e344ca391d295f92c6f6658 P 4e8bd1e7c1f848b0bdd2570de97506d8: Time spent bootstrapping tablet: real 0.040s	user 0.035s	sys 0.004s
I20240502 21:26:19.976691   516 raft_consensus.cc:348] T c2cdd0cd6e344ca391d295f92c6f6658 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } }
I20240502 21:26:19.976905   516 raft_consensus.cc:729] T c2cdd0cd6e344ca391d295f92c6f6658 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e8bd1e7c1f848b0bdd2570de97506d8, State: Initialized, Role: LEARNER
I20240502 21:26:19.977108   516 consensus_queue.cc:260] T c2cdd0cd6e344ca391d295f92c6f6658 P 4e8bd1e7c1f848b0bdd2570de97506d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } }
I20240502 21:26:19.978441   516 ts_tablet_manager.cc:1398] T c2cdd0cd6e344ca391d295f92c6f6658 P 4e8bd1e7c1f848b0bdd2570de97506d8: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 21:26:19.979430 28913 tablet_copy_service.cc:342] P 8733381e64da4e91a8a54e23b01e8967: Request end of tablet copy session 4e8bd1e7c1f848b0bdd2570de97506d8-c2cdd0cd6e344ca391d295f92c6f6658 received from {username='slave'} at 127.27.196.72:35103
I20240502 21:26:19.979605 28913 tablet_copy_service.cc:433] P 8733381e64da4e91a8a54e23b01e8967: ending tablet copy session 4e8bd1e7c1f848b0bdd2570de97506d8-c2cdd0cd6e344ca391d295f92c6f6658 on tablet c2cdd0cd6e344ca391d295f92c6f6658 with peer 4e8bd1e7c1f848b0bdd2570de97506d8
W20240502 21:26:20.107565 28824 consensus_peers.cc:485] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:26:20.500988 28687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 21:26:20.514734 30571 raft_consensus.cc:1203] T c2cdd0cd6e344ca391d295f92c6f6658 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
W20240502 21:26:20.526371 28824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 21:26:20.527711 28824 consensus_peers.cc:485] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 21:26:20.919842   595 raft_consensus.cc:1050] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967: attempting to promote NON_VOTER 4e8bd1e7c1f848b0bdd2570de97506d8 to VOTER
I20240502 21:26:20.920756   595 consensus_queue.cc:237] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } }
I20240502 21:26:20.924278 30571 raft_consensus.cc:1261] T c2cdd0cd6e344ca391d295f92c6f6658 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 LEARNER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:20.924319 30171 raft_consensus.cc:1261] T c2cdd0cd6e344ca391d295f92c6f6658 P a1e2dba8e1a3403295cb394f7090f849 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:20.924453 28756 raft_consensus.cc:1261] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Refusing update from remote peer 8733381e64da4e91a8a54e23b01e8967: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:20.925060   595 consensus_queue.cc:1035] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:20.925416   597 consensus_queue.cc:1035] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:20.925776   550 consensus_queue.cc:1035] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:20.932706   597 raft_consensus.cc:2932] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:26:20.933776 30571 raft_consensus.cc:2932] T c2cdd0cd6e344ca391d295f92c6f6658 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:26:20.933931 30171 raft_consensus.cc:2932] T c2cdd0cd6e344ca391d295f92c6f6658 P a1e2dba8e1a3403295cb394f7090f849 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:26:20.938015 28756 raft_consensus.cc:2932] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:26:20.943403 28475 catalog_manager.cc:5484] T c2cdd0cd6e344ca391d295f92c6f6658 P 4e8bd1e7c1f848b0bdd2570de97506d8 reported cstate change: config changed from index 7 to 8, 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8733381e64da4e91a8a54e23b01e8967" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:26:20.997602 28892 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c2cdd0cd6e344ca391d295f92c6f6658"
dest_uuid: "8733381e64da4e91a8a54e23b01e8967"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:45074
I20240502 21:26:20.997830 28892 raft_consensus.cc:595] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Received request to transfer leadership
W20240502 21:26:21.036830 28961 consensus_peers.cc:485] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 21:26:21.219897 28892 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c2cdd0cd6e344ca391d295f92c6f6658"
dest_uuid: "8733381e64da4e91a8a54e23b01e8967"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:45100
I20240502 21:26:21.220151 28892 raft_consensus.cc:595] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:21.400142   595 raft_consensus.cc:979] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967: : Instructing follower 8786a1419d3f4bc79b88c54ce0675aa6 to start an election
I20240502 21:26:21.400364   595 raft_consensus.cc:1067] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Signalling peer 8786a1419d3f4bc79b88c54ce0675aa6 to start an election
I20240502 21:26:21.401158 28756 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c2cdd0cd6e344ca391d295f92c6f6658"
dest_uuid: "8786a1419d3f4bc79b88c54ce0675aa6"
 from {username='slave'} at 127.27.196.67:41279
I20240502 21:26:21.401427 28756 raft_consensus.cc:482] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:21.401554 28756 raft_consensus.cc:3037] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:21.403813 28756 raft_consensus.cc:504] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } }
I20240502 21:26:21.404904 28756 leader_election.cc:288] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 2 election: Requested vote from peers 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545), a1e2dba8e1a3403295cb394f7090f849 (127.27.196.69:41545), 4e8bd1e7c1f848b0bdd2570de97506d8 (127.27.196.72:34997)
I20240502 21:26:21.405885 30571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2cdd0cd6e344ca391d295f92c6f6658" candidate_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8"
I20240502 21:26:21.405920 30171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2cdd0cd6e344ca391d295f92c6f6658" candidate_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "a1e2dba8e1a3403295cb394f7090f849"
I20240502 21:26:21.405817 28892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2cdd0cd6e344ca391d295f92c6f6658" candidate_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "8733381e64da4e91a8a54e23b01e8967"
I20240502 21:26:21.406189 30571 raft_consensus.cc:3037] T c2cdd0cd6e344ca391d295f92c6f6658 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:21.406292 30171 raft_consensus.cc:3037] T c2cdd0cd6e344ca391d295f92c6f6658 P a1e2dba8e1a3403295cb394f7090f849 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:21.406358 28892 raft_consensus.cc:3032] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:26:21.406512 28892 raft_consensus.cc:729] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8733381e64da4e91a8a54e23b01e8967, State: Running, Role: LEADER
I20240502 21:26:21.406913 28892 consensus_queue.cc:260] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } }
I20240502 21:26:21.407537 28892 raft_consensus.cc:3037] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:21.409085 30171 raft_consensus.cc:2454] T c2cdd0cd6e344ca391d295f92c6f6658 P a1e2dba8e1a3403295cb394f7090f849 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8786a1419d3f4bc79b88c54ce0675aa6 in term 2.
I20240502 21:26:21.409765 30571 raft_consensus.cc:2454] T c2cdd0cd6e344ca391d295f92c6f6658 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8786a1419d3f4bc79b88c54ce0675aa6 in term 2.
I20240502 21:26:21.410598 28686 leader_election.cc:302] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4e8bd1e7c1f848b0bdd2570de97506d8, 8786a1419d3f4bc79b88c54ce0675aa6, a1e2dba8e1a3403295cb394f7090f849; no voters: 
I20240502 21:26:21.410809 28892 raft_consensus.cc:2454] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8786a1419d3f4bc79b88c54ce0675aa6 in term 2.
I20240502 21:26:21.411382 32703 raft_consensus.cc:2781] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:26:21.411974 32703 raft_consensus.cc:686] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 2 LEADER]: Becoming Leader. State: Replica: 8786a1419d3f4bc79b88c54ce0675aa6, State: Running, Role: LEADER
I20240502 21:26:21.412508 32703 consensus_queue.cc:237] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } }
I20240502 21:26:21.416148 28472 catalog_manager.cc:5484] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 reported cstate change: term changed from 1 to 2, leader changed from 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67) to 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66). New cstate: current_term: 2 leader_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 21:26:21.539794 28687 consensus_peers.cc:485] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 21:26:21.853246 28892 raft_consensus.cc:1261] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [term 2 FOLLOWER]: Refusing update from remote peer 8786a1419d3f4bc79b88c54ce0675aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:26:21.854307 32703 consensus_queue.cc:1035] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:26:21.873893 30171 raft_consensus.cc:1261] T c2cdd0cd6e344ca391d295f92c6f6658 P a1e2dba8e1a3403295cb394f7090f849 [term 2 FOLLOWER]: Refusing update from remote peer 8786a1419d3f4bc79b88c54ce0675aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:26:21.874771 32703 consensus_queue.cc:1035] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:26:21.880503 30571 raft_consensus.cc:1261] T c2cdd0cd6e344ca391d295f92c6f6658 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 2 FOLLOWER]: Refusing update from remote peer 8786a1419d3f4bc79b88c54ce0675aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:26:21.881368   645 consensus_queue.cc:1035] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 21:26:22.013971 28824 consensus_peers.cc:485] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:26:22.148059 28687 consensus_peers.cc:485] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 21:26:22.354696 28756 consensus_queue.cc:237] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } }
I20240502 21:26:22.356645 30571 raft_consensus.cc:1261] T c2cdd0cd6e344ca391d295f92c6f6658 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 2 FOLLOWER]: Refusing update from remote peer 8786a1419d3f4bc79b88c54ce0675aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 21:26:22.356761 30171 raft_consensus.cc:1261] T c2cdd0cd6e344ca391d295f92c6f6658 P a1e2dba8e1a3403295cb394f7090f849 [term 2 FOLLOWER]: Refusing update from remote peer 8786a1419d3f4bc79b88c54ce0675aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 21:26:22.357375   645 consensus_queue.cc:1035] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:26:22.357726 32703 consensus_queue.cc:1035] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:26:22.360260 32703 raft_consensus.cc:2932] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:26:22.362567 30171 raft_consensus.cc:2932] T c2cdd0cd6e344ca391d295f92c6f6658 P a1e2dba8e1a3403295cb394f7090f849 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:26:22.364102 28461 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c2cdd0cd6e344ca391d295f92c6f6658 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:22.365250 30571 raft_consensus.cc:2932] T c2cdd0cd6e344ca391d295f92c6f6658 P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } } }
I20240502 21:26:22.365427 28467 catalog_manager.cc:5484] T c2cdd0cd6e344ca391d295f92c6f6658 P 8786a1419d3f4bc79b88c54ce0675aa6 reported cstate change: config changed from index 8 to 10, VOTER 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67) evicted. New cstate: current_term: 2 leader_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1e2dba8e1a3403295cb394f7090f849" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 41545 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:22.372252 28871 tablet_service.cc:1514] Processing DeleteTablet for tablet c2cdd0cd6e344ca391d295f92c6f6658 with delete_type TABLET_DATA_TOMBSTONED (TS 8733381e64da4e91a8a54e23b01e8967 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:53992
I20240502 21:26:22.373991   665 tablet_replica.cc:330] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967: stopping tablet replica
I20240502 21:26:22.374163   665 raft_consensus.cc:2229] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:22.374364   665 raft_consensus.cc:2258] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:22.375264   665 ts_tablet_manager.cc:1875] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:22.380931   665 ts_tablet_manager.cc:1888] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 21:26:22.381050   665 log.cc:1196] T c2cdd0cd6e344ca391d295f92c6f6658 P 8733381e64da4e91a8a54e23b01e8967: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/wals/c2cdd0cd6e344ca391d295f92c6f6658
I20240502 21:26:22.381886 28460 catalog_manager.cc:4830] TS 8733381e64da4e91a8a54e23b01e8967 (127.27.196.67:42545): tablet c2cdd0cd6e344ca391d295f92c6f6658 (table rebalance_test_table_1 [id=a4cb53742e604e37ae1a6151409c162b]) successfully deleted
I20240502 21:26:22.662940 28735 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:26:22.663065 29006 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:22.664369 28871 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:26:22.676970 30152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:22.677244 30552 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:22.682687 30420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:22.688542 30285 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:26:22.723098 28961 consensus_peers.cc:485] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:26:22.732272 28824 consensus_peers.cc:485] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:26:22.819070 28461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240502 21:26:22.822209 28461 catalog_manager.cc:4573] TS 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a2fd105773d449da89940d3b6558361a: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
W20240502 21:26:22.843492 28824 consensus_peers.cc:485] T dd7a03f1bc784533bbc10ade78700ba8 P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 21:26:22.919250 30552 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:22.943135 29006 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:22.948244 28735 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:26:22.963632 28871 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:26:22.967386 30152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:22.997666 30420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:23.011224 30285 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:23.296629 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.65:44983
--local_ip_for_outbound_sockets=127.27.196.65
--tserver_master_addrs=127.27.196.126:39011
--webserver_port=36389
--webserver_interface=127.27.196.65
--builtin_ntp_servers=127.27.196.84:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 21:26:23.329123 28961 consensus_peers.cc:485] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:26:23.710744   747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:23.711076   747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:23.711237   747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:23.719892   747 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:23.720055   747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:23.720233   747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.65
I20240502 21:26:23.730173   747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.65:44983
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.27.196.65
--webserver_port=36389
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.27.196.126:39011
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.65
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:23.732493   747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:23.734411   747 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:23.749465   752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:23.749472   753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:23.749974   755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:23.750735   747 server_base.cc:1034] running on GCE node
I20240502 21:26:23.751621   747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:23.752830   747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:23.754184   747 hybrid_clock.cc:648] HybridClock initialized: now 1714685183754107 us; error 68 us; skew 500 ppm
I20240502 21:26:23.754539   747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:23.757277   747 webserver.cc:460] Webserver started at http://127.27.196.65:36389/ using document root <none> and password file <none>
I20240502 21:26:23.758090   747 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:23.758219   747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:23.764906   747 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240502 21:26:23.771301   761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:23.773909   747 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.000s
I20240502 21:26:23.774228   747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "04e36797791842e489aee30de261c4c7"
format_stamp: "Formatted at 2024-05-02 21:25:21 on dist-test-slave-3bs1"
I20240502 21:26:23.776175   747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:23.801803   747 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:26:23.802709   747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:23.803009   747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:23.804217   747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:23.806959   768 ts_tablet_manager.cc:522] Loading tablet metadata (0/12 complete)
I20240502 21:26:23.839392   747 ts_tablet_manager.cc:559] Loaded tablet metadata (12 total tablets, 12 live tablets)
I20240502 21:26:23.839521   747 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.033s	user 0.002s	sys 0.000s
I20240502 21:26:23.839656   747 ts_tablet_manager.cc:574] Registering tablets (0/12 complete)
I20240502 21:26:23.843425   768 tablet_bootstrap.cc:492] T a2fd105773d449da89940d3b6558361a P 04e36797791842e489aee30de261c4c7: Bootstrap starting.
I20240502 21:26:23.873667   768 log.cc:824] T a2fd105773d449da89940d3b6558361a P 04e36797791842e489aee30de261c4c7: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:23.875824   747 ts_tablet_manager.cc:590] Registered 12 tablets
I20240502 21:26:23.875969   747 ts_tablet_manager.cc:569] Time spent register tablets: real 0.036s	user 0.032s	sys 0.004s
I20240502 21:26:23.945398   768 tablet_bootstrap.cc:492] T a2fd105773d449da89940d3b6558361a P 04e36797791842e489aee30de261c4c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:23.946346   768 tablet_bootstrap.cc:492] T a2fd105773d449da89940d3b6558361a P 04e36797791842e489aee30de261c4c7: Bootstrap complete.
I20240502 21:26:23.948184   768 ts_tablet_manager.cc:1367] T a2fd105773d449da89940d3b6558361a P 04e36797791842e489aee30de261c4c7: Time spent bootstrapping tablet: real 0.105s	user 0.084s	sys 0.019s
I20240502 21:26:23.957273   768 raft_consensus.cc:348] T a2fd105773d449da89940d3b6558361a P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } }
I20240502 21:26:23.958969   768 raft_consensus.cc:729] T a2fd105773d449da89940d3b6558361a P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e36797791842e489aee30de261c4c7, State: Initialized, Role: FOLLOWER
I20240502 21:26:23.960137   768 consensus_queue.cc:260] T a2fd105773d449da89940d3b6558361a P 04e36797791842e489aee30de261c4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } }
I20240502 21:26:23.961941   768 ts_tablet_manager.cc:1398] T a2fd105773d449da89940d3b6558361a P 04e36797791842e489aee30de261c4c7: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.005s
I20240502 21:26:23.962406   768 tablet_bootstrap.cc:492] T 20322b184e2f4a88b1abb569e6916f7c P 04e36797791842e489aee30de261c4c7: Bootstrap starting.
I20240502 21:26:23.974603   877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.65:44983 every 8 connection(s)
I20240502 21:26:23.974604   747 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.65:44983
I20240502 21:26:23.976979   747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:26:23.985123 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 747
I20240502 21:26:23.995509   878 heartbeater.cc:347] Connected to a master server at 127.27.196.126:39011
I20240502 21:26:23.995895   878 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:23.997167   878 heartbeater.cc:511] Master 127.27.196.126:39011 requested a full tablet report, sending...
I20240502 21:26:24.001950 28475 ts_manager.cc:194] Re-registered known tserver with Master: 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983)
I20240502 21:26:24.008062 28475 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.65:57943
I20240502 21:26:24.013422   768 tablet_bootstrap.cc:492] T 20322b184e2f4a88b1abb569e6916f7c P 04e36797791842e489aee30de261c4c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:24.013983   768 tablet_bootstrap.cc:492] T 20322b184e2f4a88b1abb569e6916f7c P 04e36797791842e489aee30de261c4c7: Bootstrap complete.
I20240502 21:26:24.015025   768 ts_tablet_manager.cc:1367] T 20322b184e2f4a88b1abb569e6916f7c P 04e36797791842e489aee30de261c4c7: Time spent bootstrapping tablet: real 0.053s	user 0.037s	sys 0.011s
I20240502 21:26:24.015928   768 raft_consensus.cc:348] T 20322b184e2f4a88b1abb569e6916f7c P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:26:24.016152   768 raft_consensus.cc:729] T 20322b184e2f4a88b1abb569e6916f7c P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e36797791842e489aee30de261c4c7, State: Initialized, Role: FOLLOWER
I20240502 21:26:24.016419   768 consensus_queue.cc:260] T 20322b184e2f4a88b1abb569e6916f7c P 04e36797791842e489aee30de261c4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:26:24.016993   768 ts_tablet_manager.cc:1398] T 20322b184e2f4a88b1abb569e6916f7c P 04e36797791842e489aee30de261c4c7: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 21:26:24.017074   878 heartbeater.cc:503] Master 127.27.196.126:39011 was elected leader, sending a full tablet report...
I20240502 21:26:24.017297   768 tablet_bootstrap.cc:492] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7: Bootstrap starting.
I20240502 21:26:24.058622   768 tablet_bootstrap.cc:492] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:24.059250   768 tablet_bootstrap.cc:492] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7: Bootstrap complete.
I20240502 21:26:24.060309   768 ts_tablet_manager.cc:1367] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7: Time spent bootstrapping tablet: real 0.043s	user 0.022s	sys 0.019s
I20240502 21:26:24.061005   768 raft_consensus.cc:348] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:26:24.061197   768 raft_consensus.cc:729] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e36797791842e489aee30de261c4c7, State: Initialized, Role: FOLLOWER
I20240502 21:26:24.061468   768 consensus_queue.cc:260] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:26:24.061991   768 ts_tablet_manager.cc:1398] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 21:26:24.062250   768 tablet_bootstrap.cc:492] T dd7a03f1bc784533bbc10ade78700ba8 P 04e36797791842e489aee30de261c4c7: Bootstrap starting.
I20240502 21:26:24.107069   768 tablet_bootstrap.cc:492] T dd7a03f1bc784533bbc10ade78700ba8 P 04e36797791842e489aee30de261c4c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:24.107798   768 tablet_bootstrap.cc:492] T dd7a03f1bc784533bbc10ade78700ba8 P 04e36797791842e489aee30de261c4c7: Bootstrap complete.
I20240502 21:26:24.109122   768 ts_tablet_manager.cc:1367] T dd7a03f1bc784533bbc10ade78700ba8 P 04e36797791842e489aee30de261c4c7: Time spent bootstrapping tablet: real 0.047s	user 0.033s	sys 0.011s
I20240502 21:26:24.110008   768 raft_consensus.cc:348] T dd7a03f1bc784533bbc10ade78700ba8 P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:26:24.110282   768 raft_consensus.cc:729] T dd7a03f1bc784533bbc10ade78700ba8 P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e36797791842e489aee30de261c4c7, State: Initialized, Role: FOLLOWER
I20240502 21:26:24.110564   768 consensus_queue.cc:260] T dd7a03f1bc784533bbc10ade78700ba8 P 04e36797791842e489aee30de261c4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:26:24.111088   768 ts_tablet_manager.cc:1398] T dd7a03f1bc784533bbc10ade78700ba8 P 04e36797791842e489aee30de261c4c7: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240502 21:26:24.111526   768 tablet_bootstrap.cc:492] T 876004fec43e428e930f3fbb873ccac8 P 04e36797791842e489aee30de261c4c7: Bootstrap starting.
I20240502 21:26:24.158879   768 tablet_bootstrap.cc:492] T 876004fec43e428e930f3fbb873ccac8 P 04e36797791842e489aee30de261c4c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:24.159560   768 tablet_bootstrap.cc:492] T 876004fec43e428e930f3fbb873ccac8 P 04e36797791842e489aee30de261c4c7: Bootstrap complete.
I20240502 21:26:24.160980   768 ts_tablet_manager.cc:1367] T 876004fec43e428e930f3fbb873ccac8 P 04e36797791842e489aee30de261c4c7: Time spent bootstrapping tablet: real 0.050s	user 0.033s	sys 0.014s
I20240502 21:26:24.162006   768 raft_consensus.cc:348] T 876004fec43e428e930f3fbb873ccac8 P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:26:24.162281   768 raft_consensus.cc:729] T 876004fec43e428e930f3fbb873ccac8 P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e36797791842e489aee30de261c4c7, State: Initialized, Role: FOLLOWER
I20240502 21:26:24.162564   768 consensus_queue.cc:260] T 876004fec43e428e930f3fbb873ccac8 P 04e36797791842e489aee30de261c4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:26:24.163352   768 ts_tablet_manager.cc:1398] T 876004fec43e428e930f3fbb873ccac8 P 04e36797791842e489aee30de261c4c7: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 21:26:24.163797   768 tablet_bootstrap.cc:492] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7: Bootstrap starting.
W20240502 21:26:24.174559 28687 consensus_peers.cc:485] T a604aabc523640089bc54cd6473a9a95 P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 81: this message will repeat every 5th retry.
I20240502 21:26:24.214301   768 tablet_bootstrap.cc:492] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:24.214874   768 tablet_bootstrap.cc:492] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7: Bootstrap complete.
I20240502 21:26:24.215983   768 ts_tablet_manager.cc:1367] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7: Time spent bootstrapping tablet: real 0.052s	user 0.031s	sys 0.020s
I20240502 21:26:24.216796   768 raft_consensus.cc:348] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:26:24.217054   768 raft_consensus.cc:729] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e36797791842e489aee30de261c4c7, State: Initialized, Role: FOLLOWER
I20240502 21:26:24.217424   768 consensus_queue.cc:260] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:26:24.217958   768 ts_tablet_manager.cc:1398] T 3665b7b9141b4fe8965eeecea35e9c0b P 04e36797791842e489aee30de261c4c7: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240502 21:26:24.218254   768 tablet_bootstrap.cc:492] T e6387ed203114433b587600b1a8a682a P 04e36797791842e489aee30de261c4c7: Bootstrap starting.
I20240502 21:26:24.267163   768 tablet_bootstrap.cc:492] T e6387ed203114433b587600b1a8a682a P 04e36797791842e489aee30de261c4c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:24.267688   768 tablet_bootstrap.cc:492] T e6387ed203114433b587600b1a8a682a P 04e36797791842e489aee30de261c4c7: Bootstrap complete.
I20240502 21:26:24.268752   768 ts_tablet_manager.cc:1367] T e6387ed203114433b587600b1a8a682a P 04e36797791842e489aee30de261c4c7: Time spent bootstrapping tablet: real 0.051s	user 0.029s	sys 0.019s
I20240502 21:26:24.269528   768 raft_consensus.cc:348] T e6387ed203114433b587600b1a8a682a P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:26:24.269841   768 raft_consensus.cc:729] T e6387ed203114433b587600b1a8a682a P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e36797791842e489aee30de261c4c7, State: Initialized, Role: FOLLOWER
I20240502 21:26:24.270216   768 consensus_queue.cc:260] T e6387ed203114433b587600b1a8a682a P 04e36797791842e489aee30de261c4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:26:24.270924   768 ts_tablet_manager.cc:1398] T e6387ed203114433b587600b1a8a682a P 04e36797791842e489aee30de261c4c7: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240502 21:26:24.271400   768 tablet_bootstrap.cc:492] T a4c6f9c73a2d4b01a2c027504e45b238 P 04e36797791842e489aee30de261c4c7: Bootstrap starting.
I20240502 21:26:24.346369   768 tablet_bootstrap.cc:492] T a4c6f9c73a2d4b01a2c027504e45b238 P 04e36797791842e489aee30de261c4c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:24.347043   768 tablet_bootstrap.cc:492] T a4c6f9c73a2d4b01a2c027504e45b238 P 04e36797791842e489aee30de261c4c7: Bootstrap complete.
I20240502 21:26:24.348456   768 ts_tablet_manager.cc:1367] T a4c6f9c73a2d4b01a2c027504e45b238 P 04e36797791842e489aee30de261c4c7: Time spent bootstrapping tablet: real 0.077s	user 0.067s	sys 0.008s
I20240502 21:26:24.349398   768 raft_consensus.cc:348] T a4c6f9c73a2d4b01a2c027504e45b238 P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } }
I20240502 21:26:24.349711   768 raft_consensus.cc:729] T a4c6f9c73a2d4b01a2c027504e45b238 P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e36797791842e489aee30de261c4c7, State: Initialized, Role: FOLLOWER
I20240502 21:26:24.350134   768 consensus_queue.cc:260] T a4c6f9c73a2d4b01a2c027504e45b238 P 04e36797791842e489aee30de261c4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } attrs { replace: true } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: NON_VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: true } }
I20240502 21:26:24.356058   768 ts_tablet_manager.cc:1398] T a4c6f9c73a2d4b01a2c027504e45b238 P 04e36797791842e489aee30de261c4c7: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.000s
I20240502 21:26:24.356833   768 tablet_bootstrap.cc:492] T e94280a60b6d4902b866ae3c30e40470 P 04e36797791842e489aee30de261c4c7: Bootstrap starting.
I20240502 21:26:24.410661   768 tablet_bootstrap.cc:492] T e94280a60b6d4902b866ae3c30e40470 P 04e36797791842e489aee30de261c4c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:24.411237   768 tablet_bootstrap.cc:492] T e94280a60b6d4902b866ae3c30e40470 P 04e36797791842e489aee30de261c4c7: Bootstrap complete.
I20240502 21:26:24.412369   768 ts_tablet_manager.cc:1367] T e94280a60b6d4902b866ae3c30e40470 P 04e36797791842e489aee30de261c4c7: Time spent bootstrapping tablet: real 0.056s	user 0.042s	sys 0.011s
I20240502 21:26:24.413125   768 raft_consensus.cc:348] T e94280a60b6d4902b866ae3c30e40470 P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:26:24.413334   768 raft_consensus.cc:729] T e94280a60b6d4902b866ae3c30e40470 P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e36797791842e489aee30de261c4c7, State: Initialized, Role: FOLLOWER
I20240502 21:26:24.413599   768 consensus_queue.cc:260] T e94280a60b6d4902b866ae3c30e40470 P 04e36797791842e489aee30de261c4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } }
I20240502 21:26:24.414181   768 ts_tablet_manager.cc:1398] T e94280a60b6d4902b866ae3c30e40470 P 04e36797791842e489aee30de261c4c7: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 21:26:24.414506   768 tablet_bootstrap.cc:492] T 4a58681ae3e54019aefa430aaa3dc046 P 04e36797791842e489aee30de261c4c7: Bootstrap starting.
I20240502 21:26:24.462633   768 tablet_bootstrap.cc:492] T 4a58681ae3e54019aefa430aaa3dc046 P 04e36797791842e489aee30de261c4c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:24.463238   768 tablet_bootstrap.cc:492] T 4a58681ae3e54019aefa430aaa3dc046 P 04e36797791842e489aee30de261c4c7: Bootstrap complete.
I20240502 21:26:24.464558   768 ts_tablet_manager.cc:1367] T 4a58681ae3e54019aefa430aaa3dc046 P 04e36797791842e489aee30de261c4c7: Time spent bootstrapping tablet: real 0.050s	user 0.021s	sys 0.027s
I20240502 21:26:24.465132   832 raft_consensus.cc:3037] T d25783a912884d2ba72c586d6cd7dfdd P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:24.465507   768 raft_consensus.cc:348] T 4a58681ae3e54019aefa430aaa3dc046 P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:26:24.465862   768 raft_consensus.cc:729] T 4a58681ae3e54019aefa430aaa3dc046 P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e36797791842e489aee30de261c4c7, State: Initialized, Role: FOLLOWER
I20240502 21:26:24.466197   768 consensus_queue.cc:260] T 4a58681ae3e54019aefa430aaa3dc046 P 04e36797791842e489aee30de261c4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } }
I20240502 21:26:24.466840   768 ts_tablet_manager.cc:1398] T 4a58681ae3e54019aefa430aaa3dc046 P 04e36797791842e489aee30de261c4c7: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240502 21:26:24.467307   768 tablet_bootstrap.cc:492] T a604aabc523640089bc54cd6473a9a95 P 04e36797791842e489aee30de261c4c7: Bootstrap starting.
I20240502 21:26:24.522318   768 tablet_bootstrap.cc:492] T a604aabc523640089bc54cd6473a9a95 P 04e36797791842e489aee30de261c4c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:24.522940   768 tablet_bootstrap.cc:492] T a604aabc523640089bc54cd6473a9a95 P 04e36797791842e489aee30de261c4c7: Bootstrap complete.
I20240502 21:26:24.524240   768 ts_tablet_manager.cc:1367] T a604aabc523640089bc54cd6473a9a95 P 04e36797791842e489aee30de261c4c7: Time spent bootstrapping tablet: real 0.057s	user 0.039s	sys 0.012s
I20240502 21:26:24.525746   768 raft_consensus.cc:348] T a604aabc523640089bc54cd6473a9a95 P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:26:24.526043   768 raft_consensus.cc:729] T a604aabc523640089bc54cd6473a9a95 P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e36797791842e489aee30de261c4c7, State: Initialized, Role: FOLLOWER
I20240502 21:26:24.526326   768 consensus_queue.cc:260] T a604aabc523640089bc54cd6473a9a95 P 04e36797791842e489aee30de261c4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "52c5becf6262415cab0408b3968ed4c0" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 41795 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:26:24.527499   768 ts_tablet_manager.cc:1398] T a604aabc523640089bc54cd6473a9a95 P 04e36797791842e489aee30de261c4c7: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 21:26:24.527781   768 tablet_bootstrap.cc:492] T 82338a2c110b44efa16b202c0bad760c P 04e36797791842e489aee30de261c4c7: Bootstrap starting.
W20240502 21:26:24.533419 28687 consensus_peers.cc:485] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 81: this message will repeat every 5th retry.
I20240502 21:26:24.585505   768 tablet_bootstrap.cc:492] T 82338a2c110b44efa16b202c0bad760c P 04e36797791842e489aee30de261c4c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:24.586148   768 tablet_bootstrap.cc:492] T 82338a2c110b44efa16b202c0bad760c P 04e36797791842e489aee30de261c4c7: Bootstrap complete.
I20240502 21:26:24.587406   768 ts_tablet_manager.cc:1367] T 82338a2c110b44efa16b202c0bad760c P 04e36797791842e489aee30de261c4c7: Time spent bootstrapping tablet: real 0.060s	user 0.051s	sys 0.007s
I20240502 21:26:24.588320   768 raft_consensus.cc:348] T 82338a2c110b44efa16b202c0bad760c P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:26:24.588577   768 raft_consensus.cc:729] T 82338a2c110b44efa16b202c0bad760c P 04e36797791842e489aee30de261c4c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e36797791842e489aee30de261c4c7, State: Initialized, Role: FOLLOWER
I20240502 21:26:24.588881   768 consensus_queue.cc:260] T 82338a2c110b44efa16b202c0bad760c P 04e36797791842e489aee30de261c4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e36797791842e489aee30de261c4c7" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 44983 } } peers { permanent_uuid: "8733381e64da4e91a8a54e23b01e8967" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 42545 } } peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } }
I20240502 21:26:24.589509   768 ts_tablet_manager.cc:1398] T 82338a2c110b44efa16b202c0bad760c P 04e36797791842e489aee30de261c4c7: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 21:26:24.662565 30552 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:24.663753   811 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:24.685703 28871 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:26:24.687188 29006 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:24.702050 28735 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:26:24.731432 30420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:24.738205 30152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:24.753083 30285 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:25.022495   811 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:25.036558 30552 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:25.047924 29006 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:25.054558 28871 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:26:25.082892 28735 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:26:25.098517 30152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:25.140221 30420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:25.144346 30285 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:25.321660 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 747
I20240502 21:26:25.329632   873 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:25.461359 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 747
W20240502 21:26:25.490537 28687 connection.cc:535] client connection to 127.27.196.65:44983 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:26:25.491044 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 28671
W20240502 21:26:25.494100 28687 consensus_peers.cc:485] T 82338a2c110b44efa16b202c0bad760c P 8786a1419d3f4bc79b88c54ce0675aa6 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:25.498597 28796 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:26:25.594189 28824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 21:26:25.595831 28824 consensus_peers.cc:485] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:26:25.595981 28824 consensus_peers.cc:485] T e6387ed203114433b587600b1a8a682a P 8733381e64da4e91a8a54e23b01e8967 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:26:25.692641 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 16 similar messages]
W20240502 21:26:25.694039 28961 consensus_peers.cc:485] T e94280a60b6d4902b866ae3c30e40470 P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:26:25.707589 28687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111) [suppressed 14 similar messages]
I20240502 21:26:25.708458 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 28671
I20240502 21:26:25.746346 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 28807
W20240502 21:26:25.746738 28824 consensus_peers.cc:485] T 20322b184e2f4a88b1abb569e6916f7c P 8733381e64da4e91a8a54e23b01e8967 -> Peer 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815): Couldn't send request to peer 8786a1419d3f4bc79b88c54ce0675aa6. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.66:41815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:25.754254 28933 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:26:25.758795 28961 consensus_peers.cc:485] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 -> Peer 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): Couldn't send request to peer 04e36797791842e489aee30de261c4c7. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:26:25.892478 28961 consensus_peers.cc:485] T d25783a912884d2ba72c586d6cd7dfdd P 52c5becf6262415cab0408b3968ed4c0 -> Peer 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815): Couldn't send request to peer 8786a1419d3f4bc79b88c54ce0675aa6. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.66:41815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:25.996500 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 28807
I20240502 21:26:26.036042 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 28944
I20240502 21:26:26.045074 29069 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:26:26.248364 28461 catalog_manager.cc:4573] TS 04e36797791842e489aee30de261c4c7 (127.27.196.65:44983): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4a58681ae3e54019aefa430aaa3dc046: Network error: Client connection negotiation failed: client connection to 127.27.196.65:44983: connect: Connection refused (error 111)
I20240502 21:26:26.261169 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 28944
W20240502 21:26:26.299165 28463 connection.cc:535] server connection from 127.27.196.68:43611 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:26:26.299407 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 30087
I20240502 21:26:26.307732 30212 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:26.449748 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 30087
W20240502 21:26:26.485896 28463 connection.cc:535] server connection from 127.27.196.69:58885 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:26:26.486189 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 30222
I20240502 21:26:26.494762 30348 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:26.657016 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 30222
W20240502 21:26:26.683064 28463 connection.cc:535] server connection from 127.27.196.70:60047 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:26:26.683473 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 30357
I20240502 21:26:26.692198 30479 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:26.816850  1015 raft_consensus.cc:482] T 3665b7b9141b4fe8965eeecea35e9c0b P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8786a1419d3f4bc79b88c54ce0675aa6)
I20240502 21:26:26.817107  1015 raft_consensus.cc:504] T 3665b7b9141b4fe8965eeecea35e9c0b P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8786a1419d3f4bc79b88c54ce0675aa6" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41815 } } peers { permanent_uuid: "fc5aa2dc1f7b4554906785d693e272d0" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 43969 } attrs { promote: false } } peers { permanent_uuid: "4e8bd1e7c1f848b0bdd2570de97506d8" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 34997 } attrs { promote: false } }
I20240502 21:26:26.818845  1015 leader_election.cc:288] T 3665b7b9141b4fe8965eeecea35e9c0b P 4e8bd1e7c1f848b0bdd2570de97506d8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815), fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70:43969)
W20240502 21:26:26.820380 30506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.66:41815: connect: Connection refused (error 111)
W20240502 21:26:26.821944 30508 leader_election.cc:334] T 3665b7b9141b4fe8965eeecea35e9c0b P 4e8bd1e7c1f848b0bdd2570de97506d8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fc5aa2dc1f7b4554906785d693e272d0 (127.27.196.70:43969): Network error: Client connection negotiation failed: client connection to 127.27.196.70:43969: connect: Connection refused (error 111)
W20240502 21:26:26.822346 30506 leader_election.cc:334] T 3665b7b9141b4fe8965eeecea35e9c0b P 4e8bd1e7c1f848b0bdd2570de97506d8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8786a1419d3f4bc79b88c54ce0675aa6 (127.27.196.66:41815): Network error: Client connection negotiation failed: client connection to 127.27.196.66:41815: connect: Connection refused (error 111)
I20240502 21:26:26.822676 30506 leader_election.cc:302] T 3665b7b9141b4fe8965eeecea35e9c0b P 4e8bd1e7c1f848b0bdd2570de97506d8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e8bd1e7c1f848b0bdd2570de97506d8; no voters: 8786a1419d3f4bc79b88c54ce0675aa6, fc5aa2dc1f7b4554906785d693e272d0
I20240502 21:26:26.823274  1015 raft_consensus.cc:2726] T 3665b7b9141b4fe8965eeecea35e9c0b P 4e8bd1e7c1f848b0bdd2570de97506d8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 21:26:26.856102 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 30357
I20240502 21:26:26.886663 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 30490
I20240502 21:26:26.894088 30612 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:27.048054 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 30490
I20240502 21:26:27.078516 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 28445
I20240502 21:26:27.080231 28507 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:27.285719 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 28445
2024-05-02T21:26:27Z chronyd exiting
[       OK ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown (67502 ms)
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest (67502 ms total)

[----------] 1 test from RebalanceStartCriteriaTest
[ RUN      ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0
I20240502 21:26:27.376226 28433 test_util.cc:268] Using random seed: 1098826802
I20240502 21:26:27.378546 28433 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:26:27.378654 28433 ts_itest-base.cc:113] --------------
I20240502 21:26:27.378691 28433 ts_itest-base.cc:114] 5 tablet servers
I20240502 21:26:27.378753 28433 ts_itest-base.cc:115] 3 replicas per TS
I20240502 21:26:27.378795 28433 ts_itest-base.cc:116] --------------
2024-05-02T21:26:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:26:27Z Disabled control of system clock
I20240502 21:26:27.415958 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.196.126:34397
--webserver_interface=127.27.196.126
--webserver_port=0
--builtin_ntp_servers=127.27.196.84:37213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.196.126:34397
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240502 21:26:27.821779  1030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:27.822166  1030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:27.822327  1030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:27.830319  1030 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:27.830516  1030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:26:27.830608  1030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:27.830703  1030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:26:27.830777  1030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:26:27.840898  1030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:37213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.27.196.126:34397
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.196.126:34397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.27.196.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:27.842463  1030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:27.844502  1030 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:27.856453  1035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:27.857139  1036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:27.857795  1038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:27.858703  1030 server_base.cc:1034] running on GCE node
I20240502 21:26:27.859639  1030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:27.861337  1030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:27.862578  1030 hybrid_clock.cc:648] HybridClock initialized: now 1714685187862520 us; error 42 us; skew 500 ppm
I20240502 21:26:27.862974  1030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:27.871881  1030 webserver.cc:460] Webserver started at http://127.27.196.126:33555/ using document root <none> and password file <none>
I20240502 21:26:27.873217  1030 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:27.873433  1030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:27.873947  1030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:27.880594  1030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d9375eff23f3416aae69cc4b2029d7d1"
format_stamp: "Formatted at 2024-05-02 21:26:27 on dist-test-slave-3bs1"
I20240502 21:26:27.881897  1030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d9375eff23f3416aae69cc4b2029d7d1"
format_stamp: "Formatted at 2024-05-02 21:26:27 on dist-test-slave-3bs1"
I20240502 21:26:27.889647  1030 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.008s	sys 0.000s
I20240502 21:26:27.894342  1044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:27.896081  1030 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.000s
I20240502 21:26:27.896322  1030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d9375eff23f3416aae69cc4b2029d7d1"
format_stamp: "Formatted at 2024-05-02 21:26:27 on dist-test-slave-3bs1"
I20240502 21:26:27.896600  1030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:27.922654  1030 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:26:27.924152  1030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:27.924587  1030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:27.947999  1030 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.126:34397
I20240502 21:26:27.948096  1096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.126:34397 every 8 connection(s)
I20240502 21:26:27.949916  1030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 21:26:27.953153 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 1030
I20240502 21:26:27.953411 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:26:27.955897  1097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:27.974216  1097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9375eff23f3416aae69cc4b2029d7d1: Bootstrap starting.
I20240502 21:26:27.978914  1097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9375eff23f3416aae69cc4b2029d7d1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:27.980856  1097 log.cc:824] T 00000000000000000000000000000000 P d9375eff23f3416aae69cc4b2029d7d1: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:27.985323  1097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9375eff23f3416aae69cc4b2029d7d1: No bootstrap required, opened a new log
I20240502 21:26:27.993506  1097 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9375eff23f3416aae69cc4b2029d7d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9375eff23f3416aae69cc4b2029d7d1" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 34397 } }
I20240502 21:26:27.994024  1097 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9375eff23f3416aae69cc4b2029d7d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:27.994192  1097 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9375eff23f3416aae69cc4b2029d7d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9375eff23f3416aae69cc4b2029d7d1, State: Initialized, Role: FOLLOWER
I20240502 21:26:27.995190  1097 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9375eff23f3416aae69cc4b2029d7d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9375eff23f3416aae69cc4b2029d7d1" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 34397 } }
I20240502 21:26:27.995469  1097 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9375eff23f3416aae69cc4b2029d7d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:26:27.995654  1097 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9375eff23f3416aae69cc4b2029d7d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:26:27.995893  1097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9375eff23f3416aae69cc4b2029d7d1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:27.998502  1097 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9375eff23f3416aae69cc4b2029d7d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9375eff23f3416aae69cc4b2029d7d1" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 34397 } }
I20240502 21:26:27.999302  1097 leader_election.cc:302] T 00000000000000000000000000000000 P d9375eff23f3416aae69cc4b2029d7d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9375eff23f3416aae69cc4b2029d7d1; no voters: 
I20240502 21:26:28.000187  1097 leader_election.cc:288] T 00000000000000000000000000000000 P d9375eff23f3416aae69cc4b2029d7d1 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 21:26:28.000403  1102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9375eff23f3416aae69cc4b2029d7d1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:28.001200  1102 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9375eff23f3416aae69cc4b2029d7d1 [term 1 LEADER]: Becoming Leader. State: Replica: d9375eff23f3416aae69cc4b2029d7d1, State: Running, Role: LEADER
I20240502 21:26:28.002038  1102 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9375eff23f3416aae69cc4b2029d7d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9375eff23f3416aae69cc4b2029d7d1" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 34397 } }
I20240502 21:26:28.003247  1097 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9375eff23f3416aae69cc4b2029d7d1 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:26:28.006165  1103 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9375eff23f3416aae69cc4b2029d7d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9375eff23f3416aae69cc4b2029d7d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9375eff23f3416aae69cc4b2029d7d1" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 34397 } } }
I20240502 21:26:28.006558  1104 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9375eff23f3416aae69cc4b2029d7d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9375eff23f3416aae69cc4b2029d7d1. Latest consensus state: current_term: 1 leader_uuid: "d9375eff23f3416aae69cc4b2029d7d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9375eff23f3416aae69cc4b2029d7d1" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 34397 } } }
I20240502 21:26:28.006855  1103 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9375eff23f3416aae69cc4b2029d7d1 [sys.catalog]: This master's current role is: LEADER
I20240502 21:26:28.006886  1104 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9375eff23f3416aae69cc4b2029d7d1 [sys.catalog]: This master's current role is: LEADER
I20240502 21:26:28.010910  1109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:26:28.017693  1109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:26:28.031783  1109 catalog_manager.cc:1347] Generated new cluster ID: f7c44bddeb1242a8b3ec46a45cc44598
I20240502 21:26:28.031978  1109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:26:28.090430  1109 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:26:28.092985  1109 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:26:28.106102  1109 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d9375eff23f3416aae69cc4b2029d7d1: Generated new TSK 0
I20240502 21:26:28.107834  1109 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:26:28.114971 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.65:0
--local_ip_for_outbound_sockets=127.27.196.65
--webserver_interface=127.27.196.65
--webserver_port=0
--tserver_master_addrs=127.27.196.126:34397
--builtin_ntp_servers=127.27.196.84:37213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 21:26:28.519531  1121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:28.519889  1121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:28.520047  1121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:28.528180  1121 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:28.528350  1121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:28.528527  1121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.65
I20240502 21:26:28.538178  1121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:37213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.27.196.65
--webserver_port=0
--tserver_master_addrs=127.27.196.126:34397
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.65
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:28.539737  1121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:28.541677  1121 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:28.555678  1126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:28.555691  1129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:28.556731  1127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:28.557368  1121 server_base.cc:1034] running on GCE node
I20240502 21:26:28.558089  1121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:28.559249  1121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:28.560560  1121 hybrid_clock.cc:648] HybridClock initialized: now 1714685188560500 us; error 53 us; skew 500 ppm
I20240502 21:26:28.560938  1121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:28.563737  1121 webserver.cc:460] Webserver started at http://127.27.196.65:45351/ using document root <none> and password file <none>
I20240502 21:26:28.564576  1121 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:28.564721  1121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:28.565075  1121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:28.568015  1121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d6e951130afe4c30ba588815b7a81efb"
format_stamp: "Formatted at 2024-05-02 21:26:28 on dist-test-slave-3bs1"
I20240502 21:26:28.568905  1121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d6e951130afe4c30ba588815b7a81efb"
format_stamp: "Formatted at 2024-05-02 21:26:28 on dist-test-slave-3bs1"
I20240502 21:26:28.576858  1121 fs_manager.cc:691] Time spent creating directory manager: real 0.008s	user 0.008s	sys 0.000s
I20240502 21:26:28.581661  1135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:28.583853  1121 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.001s
I20240502 21:26:28.584116  1121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d6e951130afe4c30ba588815b7a81efb"
format_stamp: "Formatted at 2024-05-02 21:26:28 on dist-test-slave-3bs1"
I20240502 21:26:28.584380  1121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:28.617436  1121 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:26:28.618454  1121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:28.618772  1121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:28.620110  1121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:28.621928  1121 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:28.622046  1121 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:28.622169  1121 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:28.622247  1121 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:28.661059  1121 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.65:32971
I20240502 21:26:28.661135  1248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.65:32971 every 8 connection(s)
I20240502 21:26:28.662648  1121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:26:28.665211 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 1121
I20240502 21:26:28.665443 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:26:28.671039 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.66:0
--local_ip_for_outbound_sockets=127.27.196.66
--webserver_interface=127.27.196.66
--webserver_port=0
--tserver_master_addrs=127.27.196.126:34397
--builtin_ntp_servers=127.27.196.84:37213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:26:28.683513  1249 heartbeater.cc:347] Connected to a master server at 127.27.196.126:34397
I20240502 21:26:28.684104  1249 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:28.685299  1249 heartbeater.cc:511] Master 127.27.196.126:34397 requested a full tablet report, sending...
I20240502 21:26:28.688831  1061 ts_manager.cc:194] Registered new tserver with Master: d6e951130afe4c30ba588815b7a81efb (127.27.196.65:32971)
I20240502 21:26:28.692167  1061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.65:53053
W20240502 21:26:29.079375  1253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:29.079869  1253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:29.080065  1253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:29.089210  1253 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:29.089447  1253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:29.089684  1253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.66
I20240502 21:26:29.101027  1253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:37213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.27.196.66
--webserver_port=0
--tserver_master_addrs=127.27.196.126:34397
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.66
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:29.103222  1253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:29.105827  1253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:29.123409  1258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:29.123729  1259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:29.124320  1261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:29.124843  1253 server_base.cc:1034] running on GCE node
I20240502 21:26:29.125895  1253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:29.127466  1253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:29.128785  1253 hybrid_clock.cc:648] HybridClock initialized: now 1714685189128704 us; error 63 us; skew 500 ppm
I20240502 21:26:29.129447  1253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:29.133255  1253 webserver.cc:460] Webserver started at http://127.27.196.66:42067/ using document root <none> and password file <none>
I20240502 21:26:29.134400  1253 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:29.134552  1253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:29.135022  1253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:29.138823  1253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "7af47e9891d44089aefee1f5b52046f1"
format_stamp: "Formatted at 2024-05-02 21:26:29 on dist-test-slave-3bs1"
I20240502 21:26:29.139839  1253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "7af47e9891d44089aefee1f5b52046f1"
format_stamp: "Formatted at 2024-05-02 21:26:29 on dist-test-slave-3bs1"
I20240502 21:26:29.148140  1253 fs_manager.cc:691] Time spent creating directory manager: real 0.008s	user 0.007s	sys 0.001s
I20240502 21:26:29.153765  1267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:29.156178  1253 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.002s	sys 0.000s
I20240502 21:26:29.156447  1253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "7af47e9891d44089aefee1f5b52046f1"
format_stamp: "Formatted at 2024-05-02 21:26:29 on dist-test-slave-3bs1"
I20240502 21:26:29.156716  1253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:29.175103  1253 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:26:29.176216  1253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:29.176529  1253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:29.177776  1253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:29.179626  1253 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:29.180020  1253 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:29.180205  1253 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:29.180298  1253 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:29.220011  1253 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.66:37145
I20240502 21:26:29.220089  1381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.66:37145 every 8 connection(s)
I20240502 21:26:29.221963  1253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:26:29.224375 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 1253
I20240502 21:26:29.224668 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:26:29.229226 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.67:0
--local_ip_for_outbound_sockets=127.27.196.67
--webserver_interface=127.27.196.67
--webserver_port=0
--tserver_master_addrs=127.27.196.126:34397
--builtin_ntp_servers=127.27.196.84:37213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:26:29.241724  1382 heartbeater.cc:347] Connected to a master server at 127.27.196.126:34397
I20240502 21:26:29.242199  1382 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:29.243316  1382 heartbeater.cc:511] Master 127.27.196.126:34397 requested a full tablet report, sending...
I20240502 21:26:29.245442  1061 ts_manager.cc:194] Registered new tserver with Master: 7af47e9891d44089aefee1f5b52046f1 (127.27.196.66:37145)
I20240502 21:26:29.246357  1061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.66:46309
W20240502 21:26:29.618144  1386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:29.618485  1386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:29.618847  1386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:29.627863  1386 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:29.628062  1386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:29.628243  1386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.67
I20240502 21:26:29.639434  1386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:37213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.27.196.67
--webserver_port=0
--tserver_master_addrs=127.27.196.126:34397
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.67
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:29.641023  1386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:29.642887  1386 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:29.656672  1391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:29.657231  1394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:29.657791  1392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:29.658244  1386 server_base.cc:1034] running on GCE node
I20240502 21:26:29.658943  1386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:29.660081  1386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:29.661428  1386 hybrid_clock.cc:648] HybridClock initialized: now 1714685189661315 us; error 74 us; skew 500 ppm
I20240502 21:26:29.662160  1386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:29.665493  1386 webserver.cc:460] Webserver started at http://127.27.196.67:38257/ using document root <none> and password file <none>
I20240502 21:26:29.666501  1386 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:29.666651  1386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:29.667073  1386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:29.670564  1386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "4069f5c389eb4d34b180dfaae347e65e"
format_stamp: "Formatted at 2024-05-02 21:26:29 on dist-test-slave-3bs1"
I20240502 21:26:29.671684  1386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "4069f5c389eb4d34b180dfaae347e65e"
format_stamp: "Formatted at 2024-05-02 21:26:29 on dist-test-slave-3bs1"
I20240502 21:26:29.678869  1386 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.006s	sys 0.001s
I20240502 21:26:29.683420  1400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:29.685582  1386 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 21:26:29.685982  1386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "4069f5c389eb4d34b180dfaae347e65e"
format_stamp: "Formatted at 2024-05-02 21:26:29 on dist-test-slave-3bs1"
I20240502 21:26:29.686249  1386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:29.698613  1249 heartbeater.cc:503] Master 127.27.196.126:34397 was elected leader, sending a full tablet report...
I20240502 21:26:29.706605  1386 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:26:29.707674  1386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:29.707979  1386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:29.709175  1386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:29.711009  1386 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:29.711143  1386 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:29.711262  1386 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:29.711369  1386 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:29.748940  1386 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.67:37873
I20240502 21:26:29.749022  1514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.67:37873 every 8 connection(s)
I20240502 21:26:29.750454  1386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:26:29.761004 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 1386
I20240502 21:26:29.761301 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:26:29.765925 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.68:0
--local_ip_for_outbound_sockets=127.27.196.68
--webserver_interface=127.27.196.68
--webserver_port=0
--tserver_master_addrs=127.27.196.126:34397
--builtin_ntp_servers=127.27.196.84:37213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:26:29.767807  1515 heartbeater.cc:347] Connected to a master server at 127.27.196.126:34397
I20240502 21:26:29.768201  1515 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:29.769292  1515 heartbeater.cc:511] Master 127.27.196.126:34397 requested a full tablet report, sending...
I20240502 21:26:29.771641  1061 ts_manager.cc:194] Registered new tserver with Master: 4069f5c389eb4d34b180dfaae347e65e (127.27.196.67:37873)
I20240502 21:26:29.772791  1061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.67:39939
W20240502 21:26:30.144759  1519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:30.145226  1519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:30.145385  1519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:30.154837  1519 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:30.155015  1519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:30.155252  1519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.68
I20240502 21:26:30.166350  1519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:37213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.27.196.68
--webserver_port=0
--tserver_master_addrs=127.27.196.126:34397
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.68
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:30.168309  1519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:30.170727  1519 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:30.186923  1528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:30.187175  1525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:30.187708  1526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:30.188405  1519 server_base.cc:1034] running on GCE node
I20240502 21:26:30.189424  1519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:30.190810  1519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:30.192118  1519 hybrid_clock.cc:648] HybridClock initialized: now 1714685190192034 us; error 58 us; skew 500 ppm
I20240502 21:26:30.192665  1519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:30.195616  1519 webserver.cc:460] Webserver started at http://127.27.196.68:36645/ using document root <none> and password file <none>
I20240502 21:26:30.196532  1519 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:30.196673  1519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:30.197043  1519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:30.200146  1519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "5528b170c7b343bab92e15bdc8608827"
format_stamp: "Formatted at 2024-05-02 21:26:30 on dist-test-slave-3bs1"
I20240502 21:26:30.201090  1519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "5528b170c7b343bab92e15bdc8608827"
format_stamp: "Formatted at 2024-05-02 21:26:30 on dist-test-slave-3bs1"
I20240502 21:26:30.209005  1519 fs_manager.cc:691] Time spent creating directory manager: real 0.008s	user 0.008s	sys 0.000s
I20240502 21:26:30.214466  1534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:30.216389  1519 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 21:26:30.216621  1519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "5528b170c7b343bab92e15bdc8608827"
format_stamp: "Formatted at 2024-05-02 21:26:30 on dist-test-slave-3bs1"
I20240502 21:26:30.216876  1519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:30.231094  1519 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:26:30.232193  1519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:30.232518  1519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:30.234230  1519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:30.236512  1519 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:30.236655  1519 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:30.236812  1519 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:30.236898  1519 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:30.249979  1382 heartbeater.cc:503] Master 127.27.196.126:34397 was elected leader, sending a full tablet report...
I20240502 21:26:30.274792  1519 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.68:43263
I20240502 21:26:30.274873  1647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.68:43263 every 8 connection(s)
I20240502 21:26:30.276226  1519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:26:30.281456 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 1519
I20240502 21:26:30.281738 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 21:26:30.285634 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.69:0
--local_ip_for_outbound_sockets=127.27.196.69
--webserver_interface=127.27.196.69
--webserver_port=0
--tserver_master_addrs=127.27.196.126:34397
--builtin_ntp_servers=127.27.196.84:37213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:26:30.293586  1648 heartbeater.cc:347] Connected to a master server at 127.27.196.126:34397
I20240502 21:26:30.293987  1648 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:30.294891  1648 heartbeater.cc:511] Master 127.27.196.126:34397 requested a full tablet report, sending...
I20240502 21:26:30.297114  1061 ts_manager.cc:194] Registered new tserver with Master: 5528b170c7b343bab92e15bdc8608827 (127.27.196.68:43263)
I20240502 21:26:30.298379  1061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.68:50843
W20240502 21:26:30.677894  1653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:30.678223  1653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:30.678324  1653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:30.686419  1653 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:30.686638  1653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:30.686805  1653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.69
I20240502 21:26:30.698549  1653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:37213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.27.196.69
--webserver_port=0
--tserver_master_addrs=127.27.196.126:34397
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.69
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:30.700085  1653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:30.702103  1653 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:30.715986  1660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:30.716609  1659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:30.717229  1662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:30.718034  1653 server_base.cc:1034] running on GCE node
I20240502 21:26:30.718703  1653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:30.719769  1653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:30.721108  1653 hybrid_clock.cc:648] HybridClock initialized: now 1714685190721030 us; error 64 us; skew 500 ppm
I20240502 21:26:30.721505  1653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:30.724066  1653 webserver.cc:460] Webserver started at http://127.27.196.69:35669/ using document root <none> and password file <none>
I20240502 21:26:30.724844  1653 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:30.724956  1653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:30.725306  1653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:30.728210  1653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "67c2508bffc04c5c808345780ae3faba"
format_stamp: "Formatted at 2024-05-02 21:26:30 on dist-test-slave-3bs1"
I20240502 21:26:30.729035  1653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "67c2508bffc04c5c808345780ae3faba"
format_stamp: "Formatted at 2024-05-02 21:26:30 on dist-test-slave-3bs1"
I20240502 21:26:30.734663  1653 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20240502 21:26:30.738366  1668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:30.739887  1653 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 21:26:30.740063  1653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "67c2508bffc04c5c808345780ae3faba"
format_stamp: "Formatted at 2024-05-02 21:26:30 on dist-test-slave-3bs1"
I20240502 21:26:30.740279  1653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:30.753325  1653 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:26:30.754177  1653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:30.754467  1653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:30.755708  1653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:30.757309  1653 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:30.757413  1653 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:30.757542  1653 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:30.757619  1653 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:30.775913  1515 heartbeater.cc:503] Master 127.27.196.126:34397 was elected leader, sending a full tablet report...
I20240502 21:26:30.794548  1653 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.69:33085
I20240502 21:26:30.794613  1781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.69:33085 every 8 connection(s)
I20240502 21:26:30.795944  1653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:26:30.804430 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 1653
I20240502 21:26:30.804596 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 21:26:30.810976  1782 heartbeater.cc:347] Connected to a master server at 127.27.196.126:34397
I20240502 21:26:30.811414  1782 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:30.812371  1782 heartbeater.cc:511] Master 127.27.196.126:34397 requested a full tablet report, sending...
I20240502 21:26:30.814182  1061 ts_manager.cc:194] Registered new tserver with Master: 67c2508bffc04c5c808345780ae3faba (127.27.196.69:33085)
I20240502 21:26:30.815039  1061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.69:40161
I20240502 21:26:30.821763 28433 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 21:26:30.843092  1061 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35418:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240502 21:26:30.899133  1183 tablet_service.cc:1467] Processing CreateTablet for tablet dfbca62ca938474fae4c9814f6e5086e (DEFAULT_TABLE table=TestTable [id=3a562aedb15f4e99a8d98c3ab22d4142]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:30.900511  1315 tablet_service.cc:1467] Processing CreateTablet for tablet dfbca62ca938474fae4c9814f6e5086e (DEFAULT_TABLE table=TestTable [id=3a562aedb15f4e99a8d98c3ab22d4142]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:30.901186  1183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfbca62ca938474fae4c9814f6e5086e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:30.900758  1582 tablet_service.cc:1467] Processing CreateTablet for tablet dfbca62ca938474fae4c9814f6e5086e (DEFAULT_TABLE table=TestTable [id=3a562aedb15f4e99a8d98c3ab22d4142]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:30.902217  1315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfbca62ca938474fae4c9814f6e5086e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:30.902455  1582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfbca62ca938474fae4c9814f6e5086e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:30.915937  1801 tablet_bootstrap.cc:492] T dfbca62ca938474fae4c9814f6e5086e P 5528b170c7b343bab92e15bdc8608827: Bootstrap starting.
I20240502 21:26:30.920992  1801 tablet_bootstrap.cc:654] T dfbca62ca938474fae4c9814f6e5086e P 5528b170c7b343bab92e15bdc8608827: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:30.921058  1802 tablet_bootstrap.cc:492] T dfbca62ca938474fae4c9814f6e5086e P d6e951130afe4c30ba588815b7a81efb: Bootstrap starting.
I20240502 21:26:30.922397  1803 tablet_bootstrap.cc:492] T dfbca62ca938474fae4c9814f6e5086e P 7af47e9891d44089aefee1f5b52046f1: Bootstrap starting.
I20240502 21:26:30.923144  1801 log.cc:824] T dfbca62ca938474fae4c9814f6e5086e P 5528b170c7b343bab92e15bdc8608827: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:30.926096  1803 tablet_bootstrap.cc:654] T dfbca62ca938474fae4c9814f6e5086e P 7af47e9891d44089aefee1f5b52046f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:30.926324  1802 tablet_bootstrap.cc:654] T dfbca62ca938474fae4c9814f6e5086e P d6e951130afe4c30ba588815b7a81efb: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:30.926486  1801 tablet_bootstrap.cc:492] T dfbca62ca938474fae4c9814f6e5086e P 5528b170c7b343bab92e15bdc8608827: No bootstrap required, opened a new log
I20240502 21:26:30.926851  1801 ts_tablet_manager.cc:1367] T dfbca62ca938474fae4c9814f6e5086e P 5528b170c7b343bab92e15bdc8608827: Time spent bootstrapping tablet: real 0.011s	user 0.005s	sys 0.005s
I20240502 21:26:30.927538  1803 log.cc:824] T dfbca62ca938474fae4c9814f6e5086e P 7af47e9891d44089aefee1f5b52046f1: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:30.928079  1802 log.cc:824] T dfbca62ca938474fae4c9814f6e5086e P d6e951130afe4c30ba588815b7a81efb: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:30.937614  1801 raft_consensus.cc:348] T dfbca62ca938474fae4c9814f6e5086e P 5528b170c7b343bab92e15bdc8608827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7af47e9891d44089aefee1f5b52046f1" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 37145 } } peers { permanent_uuid: "d6e951130afe4c30ba588815b7a81efb" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32971 } } peers { permanent_uuid: "5528b170c7b343bab92e15bdc8608827" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 43263 } }
I20240502 21:26:30.938035  1801 raft_consensus.cc:374] T dfbca62ca938474fae4c9814f6e5086e P 5528b170c7b343bab92e15bdc8608827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:30.938155  1802 tablet_bootstrap.cc:492] T dfbca62ca938474fae4c9814f6e5086e P d6e951130afe4c30ba588815b7a81efb: No bootstrap required, opened a new log
I20240502 21:26:30.938154  1803 tablet_bootstrap.cc:492] T dfbca62ca938474fae4c9814f6e5086e P 7af47e9891d44089aefee1f5b52046f1: No bootstrap required, opened a new log
I20240502 21:26:30.938161  1801 raft_consensus.cc:729] T dfbca62ca938474fae4c9814f6e5086e P 5528b170c7b343bab92e15bdc8608827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5528b170c7b343bab92e15bdc8608827, State: Initialized, Role: FOLLOWER
I20240502 21:26:30.938508  1803 ts_tablet_manager.cc:1367] T dfbca62ca938474fae4c9814f6e5086e P 7af47e9891d44089aefee1f5b52046f1: Time spent bootstrapping tablet: real 0.016s	user 0.005s	sys 0.008s
I20240502 21:26:30.938508  1802 ts_tablet_manager.cc:1367] T dfbca62ca938474fae4c9814f6e5086e P d6e951130afe4c30ba588815b7a81efb: Time spent bootstrapping tablet: real 0.018s	user 0.010s	sys 0.003s
I20240502 21:26:30.938885  1801 consensus_queue.cc:260] T dfbca62ca938474fae4c9814f6e5086e P 5528b170c7b343bab92e15bdc8608827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7af47e9891d44089aefee1f5b52046f1" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 37145 } } peers { permanent_uuid: "d6e951130afe4c30ba588815b7a81efb" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32971 } } peers { permanent_uuid: "5528b170c7b343bab92e15bdc8608827" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 43263 } }
I20240502 21:26:30.942102  1648 heartbeater.cc:503] Master 127.27.196.126:34397 was elected leader, sending a full tablet report...
I20240502 21:26:30.943763  1801 ts_tablet_manager.cc:1398] T dfbca62ca938474fae4c9814f6e5086e P 5528b170c7b343bab92e15bdc8608827: Time spent starting tablet: real 0.017s	user 0.015s	sys 0.000s
I20240502 21:26:30.948437  1803 raft_consensus.cc:348] T dfbca62ca938474fae4c9814f6e5086e P 7af47e9891d44089aefee1f5b52046f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7af47e9891d44089aefee1f5b52046f1" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 37145 } } peers { permanent_uuid: "d6e951130afe4c30ba588815b7a81efb" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32971 } } peers { permanent_uuid: "5528b170c7b343bab92e15bdc8608827" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 43263 } }
I20240502 21:26:30.948971  1803 raft_consensus.cc:374] T dfbca62ca938474fae4c9814f6e5086e P 7af47e9891d44089aefee1f5b52046f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:30.949098  1803 raft_consensus.cc:729] T dfbca62ca938474fae4c9814f6e5086e P 7af47e9891d44089aefee1f5b52046f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af47e9891d44089aefee1f5b52046f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:30.949896  1803 consensus_queue.cc:260] T dfbca62ca938474fae4c9814f6e5086e P 7af47e9891d44089aefee1f5b52046f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7af47e9891d44089aefee1f5b52046f1" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 37145 } } peers { permanent_uuid: "d6e951130afe4c30ba588815b7a81efb" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32971 } } peers { permanent_uuid: "5528b170c7b343bab92e15bdc8608827" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 43263 } }
I20240502 21:26:30.951848  1803 ts_tablet_manager.cc:1398] T dfbca62ca938474fae4c9814f6e5086e P 7af47e9891d44089aefee1f5b52046f1: Time spent starting tablet: real 0.013s	user 0.009s	sys 0.004s
I20240502 21:26:30.952482  1802 raft_consensus.cc:348] T dfbca62ca938474fae4c9814f6e5086e P d6e951130afe4c30ba588815b7a81efb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7af47e9891d44089aefee1f5b52046f1" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 37145 } } peers { permanent_uuid: "d6e951130afe4c30ba588815b7a81efb" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32971 } } peers { permanent_uuid: "5528b170c7b343bab92e15bdc8608827" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 43263 } }
I20240502 21:26:30.953001  1802 raft_consensus.cc:374] T dfbca62ca938474fae4c9814f6e5086e P d6e951130afe4c30ba588815b7a81efb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:30.953109  1802 raft_consensus.cc:729] T dfbca62ca938474fae4c9814f6e5086e P d6e951130afe4c30ba588815b7a81efb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e951130afe4c30ba588815b7a81efb, State: Initialized, Role: FOLLOWER
I20240502 21:26:30.954015  1802 consensus_queue.cc:260] T dfbca62ca938474fae4c9814f6e5086e P d6e951130afe4c30ba588815b7a81efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7af47e9891d44089aefee1f5b52046f1" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 37145 } } peers { permanent_uuid: "d6e951130afe4c30ba588815b7a81efb" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32971 } } peers { permanent_uuid: "5528b170c7b343bab92e15bdc8608827" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 43263 } }
I20240502 21:26:30.956195  1802 ts_tablet_manager.cc:1398] T dfbca62ca938474fae4c9814f6e5086e P d6e951130afe4c30ba588815b7a81efb: Time spent starting tablet: real 0.018s	user 0.014s	sys 0.001s
I20240502 21:26:30.966286  1809 raft_consensus.cc:482] T dfbca62ca938474fae4c9814f6e5086e P d6e951130afe4c30ba588815b7a81efb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:30.966564  1809 raft_consensus.cc:504] T dfbca62ca938474fae4c9814f6e5086e P d6e951130afe4c30ba588815b7a81efb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7af47e9891d44089aefee1f5b52046f1" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 37145 } } peers { permanent_uuid: "d6e951130afe4c30ba588815b7a81efb" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32971 } } peers { permanent_uuid: "5528b170c7b343bab92e15bdc8608827" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 43263 } }
I20240502 21:26:30.968503  1809 leader_election.cc:288] T dfbca62ca938474fae4c9814f6e5086e P d6e951130afe4c30ba588815b7a81efb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7af47e9891d44089aefee1f5b52046f1 (127.27.196.66:37145), 5528b170c7b343bab92e15bdc8608827 (127.27.196.68:43263)
W20240502 21:26:30.976581  1383 tablet.cc:2369] T dfbca62ca938474fae4c9814f6e5086e P 7af47e9891d44089aefee1f5b52046f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:30.976989  1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfbca62ca938474fae4c9814f6e5086e" candidate_uuid: "d6e951130afe4c30ba588815b7a81efb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5528b170c7b343bab92e15bdc8608827" is_pre_election: true
I20240502 21:26:30.977191  1336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfbca62ca938474fae4c9814f6e5086e" candidate_uuid: "d6e951130afe4c30ba588815b7a81efb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af47e9891d44089aefee1f5b52046f1" is_pre_election: true
I20240502 21:26:30.977643  1602 raft_consensus.cc:2454] T dfbca62ca938474fae4c9814f6e5086e P 5528b170c7b343bab92e15bdc8608827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e951130afe4c30ba588815b7a81efb in term 0.
I20240502 21:26:30.978092  1336 raft_consensus.cc:2454] T dfbca62ca938474fae4c9814f6e5086e P 7af47e9891d44089aefee1f5b52046f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e951130afe4c30ba588815b7a81efb in term 0.
I20240502 21:26:30.978420  1138 leader_election.cc:302] T dfbca62ca938474fae4c9814f6e5086e P d6e951130afe4c30ba588815b7a81efb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5528b170c7b343bab92e15bdc8608827, d6e951130afe4c30ba588815b7a81efb; no voters: 
I20240502 21:26:30.979197  1809 raft_consensus.cc:2781] T dfbca62ca938474fae4c9814f6e5086e P d6e951130afe4c30ba588815b7a81efb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:30.979329  1809 raft_consensus.cc:482] T dfbca62ca938474fae4c9814f6e5086e P d6e951130afe4c30ba588815b7a81efb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:30.979431  1809 raft_consensus.cc:3037] T dfbca62ca938474fae4c9814f6e5086e P d6e951130afe4c30ba588815b7a81efb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:30.982239  1809 raft_consensus.cc:504] T dfbca62ca938474fae4c9814f6e5086e P d6e951130afe4c30ba588815b7a81efb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7af47e9891d44089aefee1f5b52046f1" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 37145 } } peers { permanent_uuid: "d6e951130afe4c30ba588815b7a81efb" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32971 } } peers { permanent_uuid: "5528b170c7b343bab92e15bdc8608827" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 43263 } }
I20240502 21:26:30.983165  1809 leader_election.cc:288] T dfbca62ca938474fae4c9814f6e5086e P d6e951130afe4c30ba588815b7a81efb [CANDIDATE]: Term 1 election: Requested vote from peers 7af47e9891d44089aefee1f5b52046f1 (127.27.196.66:37145), 5528b170c7b343bab92e15bdc8608827 (127.27.196.68:43263)
I20240502 21:26:30.984011  1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfbca62ca938474fae4c9814f6e5086e" candidate_uuid: "d6e951130afe4c30ba588815b7a81efb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5528b170c7b343bab92e15bdc8608827"
I20240502 21:26:30.984236  1336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfbca62ca938474fae4c9814f6e5086e" candidate_uuid: "d6e951130afe4c30ba588815b7a81efb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af47e9891d44089aefee1f5b52046f1"
I20240502 21:26:30.984275  1602 raft_consensus.cc:3037] T dfbca62ca938474fae4c9814f6e5086e P 5528b170c7b343bab92e15bdc8608827 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:30.984493  1336 raft_consensus.cc:3037] T dfbca62ca938474fae4c9814f6e5086e P 7af47e9891d44089aefee1f5b52046f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:30.987357  1336 raft_consensus.cc:2454] T dfbca62ca938474fae4c9814f6e5086e P 7af47e9891d44089aefee1f5b52046f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e951130afe4c30ba588815b7a81efb in term 1.
I20240502 21:26:30.987821  1602 raft_consensus.cc:2454] T dfbca62ca938474fae4c9814f6e5086e P 5528b170c7b343bab92e15bdc8608827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e951130afe4c30ba588815b7a81efb in term 1.
I20240502 21:26:30.988075  1139 leader_election.cc:302] T dfbca62ca938474fae4c9814f6e5086e P d6e951130afe4c30ba588815b7a81efb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7af47e9891d44089aefee1f5b52046f1, d6e951130afe4c30ba588815b7a81efb; no voters: 
I20240502 21:26:30.988593  1809 raft_consensus.cc:2781] T dfbca62ca938474fae4c9814f6e5086e P d6e951130afe4c30ba588815b7a81efb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:30.989255  1809 raft_consensus.cc:686] T dfbca62ca938474fae4c9814f6e5086e P d6e951130afe4c30ba588815b7a81efb [term 1 LEADER]: Becoming Leader. State: Replica: d6e951130afe4c30ba588815b7a81efb, State: Running, Role: LEADER
I20240502 21:26:30.990015  1809 consensus_queue.cc:237] T dfbca62ca938474fae4c9814f6e5086e P d6e951130afe4c30ba588815b7a81efb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7af47e9891d44089aefee1f5b52046f1" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 37145 } } peers { permanent_uuid: "d6e951130afe4c30ba588815b7a81efb" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32971 } } peers { permanent_uuid: "5528b170c7b343bab92e15bdc8608827" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 43263 } }
I20240502 21:26:30.996623  1061 catalog_manager.cc:5484] T dfbca62ca938474fae4c9814f6e5086e P d6e951130afe4c30ba588815b7a81efb reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e951130afe4c30ba588815b7a81efb (127.27.196.65). New cstate: current_term: 1 leader_uuid: "d6e951130afe4c30ba588815b7a81efb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7af47e9891d44089aefee1f5b52046f1" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 37145 } } peers { permanent_uuid: "d6e951130afe4c30ba588815b7a81efb" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32971 } } peers { permanent_uuid: "5528b170c7b343bab92e15bdc8608827" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 43263 } } }
I20240502 21:26:31.026314 28433 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
W20240502 21:26:31.028820  1649 tablet.cc:2369] T dfbca62ca938474fae4c9814f6e5086e P 5528b170c7b343bab92e15bdc8608827: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:31.030983 28433 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d6e951130afe4c30ba588815b7a81efb to finish bootstrapping
I20240502 21:26:31.043097 28433 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7af47e9891d44089aefee1f5b52046f1 to finish bootstrapping
I20240502 21:26:31.053654 28433 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4069f5c389eb4d34b180dfaae347e65e to finish bootstrapping
I20240502 21:26:31.065232 28433 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 5528b170c7b343bab92e15bdc8608827 to finish bootstrapping
I20240502 21:26:31.076071 28433 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 67c2508bffc04c5c808345780ae3faba to finish bootstrapping
W20240502 21:26:31.167443  1250 tablet.cc:2369] T dfbca62ca938474fae4c9814f6e5086e P d6e951130afe4c30ba588815b7a81efb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:31.536636  1830 consensus_queue.cc:1035] T dfbca62ca938474fae4c9814f6e5086e P d6e951130afe4c30ba588815b7a81efb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af47e9891d44089aefee1f5b52046f1" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 37145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:31.540930  1830 consensus_queue.cc:1035] T dfbca62ca938474fae4c9814f6e5086e P d6e951130afe4c30ba588815b7a81efb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5528b170c7b343bab92e15bdc8608827" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 43263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:31.685722  1449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:31.694751  1716 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:31.698477  1315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:31.699295  1582 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:31.729892  1183 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:31.817853  1782 heartbeater.cc:503] Master 127.27.196.126:34397 was elected leader, sending a full tablet report...
I20240502 21:26:31.911783  1449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:31.916229  1582 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:31.922315  1716 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:31.922714  1315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:31.931260  1183 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:32.712366  1716 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:32.712484  1582 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:32.718427  1315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:32.718602  1449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:32.738771  1183 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:32.936920  1449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:32.947466  1716 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:32.949582  1582 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:32.964879  1315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:32.978648  1183 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:33.177081  1449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:33.184120  1716 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:33.187916  1315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:33.189771  1582 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:33.193202  1183 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:33.378955 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 1121
I20240502 21:26:33.380887  1243 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:33.490940 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 1121
W20240502 21:26:33.513209  1269 connection.cc:535] server connection from 127.27.196.65:53609 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:26:33.513621 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 1253
I20240502 21:26:33.515039  1376 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:33.625856 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 1253
W20240502 21:26:33.649724  1047 connection.cc:535] server connection from 127.27.196.66:46309 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:26:33.649948 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 1386
I20240502 21:26:33.651468  1510 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:33.753687 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 1386
I20240502 21:26:33.774315 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 1519
I20240502 21:26:33.775499  1643 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:33.874996 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 1519
I20240502 21:26:33.899471 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 1653
I20240502 21:26:33.900936  1776 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:33.998677 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 1653
I20240502 21:26:34.020558 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 1030
I20240502 21:26:34.022475  1091 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:34.158443 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 1030
2024-05-02T21:26:34Z chronyd exiting
[       OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0 (6832 ms)
[----------] 1 test from RebalanceStartCriteriaTest (6832 ms total)

[----------] 1 test from RebalanceParamTest
[ RUN      ] RebalanceParamTest.Rebalance/2
I20240502 21:26:34.208684 28433 test_util.cc:268] Using random seed: 1105659280
I20240502 21:26:34.211197 28433 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:26:34.211292 28433 ts_itest-base.cc:113] --------------
I20240502 21:26:34.211387 28433 ts_itest-base.cc:114] 7 tablet servers
I20240502 21:26:34.211441 28433 ts_itest-base.cc:115] 2 replicas per TS
I20240502 21:26:34.211515 28433 ts_itest-base.cc:116] --------------
2024-05-02T21:26:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:26:34Z Disabled control of system clock
I20240502 21:26:34.245669 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.196.126:43629
--webserver_interface=127.27.196.126
--webserver_port=0
--builtin_ntp_servers=127.27.196.84:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.196.126:43629
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240502 21:26:34.632529  2021 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 21:26:34.632933  2021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:34.633026  2021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:34.633148  2021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:34.641949  2021 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:34.642108  2021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:26:34.642172  2021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:34.642230  2021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:26:34.642339  2021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:26:34.652536  2021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.27.196.126:43629
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.196.126:43629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.27.196.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:34.654093  2021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:34.656168  2021 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:34.669232  2026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:34.669924  2029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:34.670506  2021 server_base.cc:1034] running on GCE node
W20240502 21:26:34.671022  2027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:34.672199  2021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:34.673949  2021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:34.675251  2021 hybrid_clock.cc:648] HybridClock initialized: now 1714685194675185 us; error 44 us; skew 500 ppm
I20240502 21:26:34.675638  2021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:34.678390  2021 webserver.cc:460] Webserver started at http://127.27.196.126:38639/ using document root <none> and password file <none>
I20240502 21:26:34.679308  2021 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:34.679476  2021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:34.679827  2021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:34.682885  2021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b87cabdf7aa145babe0be5701e5c3815"
format_stamp: "Formatted at 2024-05-02 21:26:34 on dist-test-slave-3bs1"
I20240502 21:26:34.683848  2021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b87cabdf7aa145babe0be5701e5c3815"
format_stamp: "Formatted at 2024-05-02 21:26:34 on dist-test-slave-3bs1"
I20240502 21:26:34.690737  2021 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.000s
I20240502 21:26:34.696235  2035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:34.698318  2021 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.005s	sys 0.000s
I20240502 21:26:34.698652  2021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b87cabdf7aa145babe0be5701e5c3815"
format_stamp: "Formatted at 2024-05-02 21:26:34 on dist-test-slave-3bs1"
I20240502 21:26:34.698938  2021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:34.723765  2021 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:26:34.724792  2021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:34.725176  2021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:34.750568  2021 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.126:43629
I20240502 21:26:34.750623  2087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.126:43629 every 8 connection(s)
I20240502 21:26:34.752863  2021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 21:26:34.758108  2088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:34.761334 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 2021
I20240502 21:26:34.761570 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:26:34.771167  2088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b87cabdf7aa145babe0be5701e5c3815: Bootstrap starting.
I20240502 21:26:34.775444  2088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b87cabdf7aa145babe0be5701e5c3815: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:34.777053  2088 log.cc:824] T 00000000000000000000000000000000 P b87cabdf7aa145babe0be5701e5c3815: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:34.781584  2088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b87cabdf7aa145babe0be5701e5c3815: No bootstrap required, opened a new log
I20240502 21:26:34.788767  2088 raft_consensus.cc:348] T 00000000000000000000000000000000 P b87cabdf7aa145babe0be5701e5c3815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87cabdf7aa145babe0be5701e5c3815" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 43629 } }
I20240502 21:26:34.789271  2088 raft_consensus.cc:374] T 00000000000000000000000000000000 P b87cabdf7aa145babe0be5701e5c3815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:34.789412  2088 raft_consensus.cc:729] T 00000000000000000000000000000000 P b87cabdf7aa145babe0be5701e5c3815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87cabdf7aa145babe0be5701e5c3815, State: Initialized, Role: FOLLOWER
I20240502 21:26:34.790306  2088 consensus_queue.cc:260] T 00000000000000000000000000000000 P b87cabdf7aa145babe0be5701e5c3815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87cabdf7aa145babe0be5701e5c3815" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 43629 } }
I20240502 21:26:34.790570  2088 raft_consensus.cc:388] T 00000000000000000000000000000000 P b87cabdf7aa145babe0be5701e5c3815 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:26:34.790719  2088 raft_consensus.cc:482] T 00000000000000000000000000000000 P b87cabdf7aa145babe0be5701e5c3815 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:26:34.790879  2088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b87cabdf7aa145babe0be5701e5c3815 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:34.793246  2088 raft_consensus.cc:504] T 00000000000000000000000000000000 P b87cabdf7aa145babe0be5701e5c3815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87cabdf7aa145babe0be5701e5c3815" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 43629 } }
I20240502 21:26:34.793877  2088 leader_election.cc:302] T 00000000000000000000000000000000 P b87cabdf7aa145babe0be5701e5c3815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b87cabdf7aa145babe0be5701e5c3815; no voters: 
I20240502 21:26:34.794716  2088 leader_election.cc:288] T 00000000000000000000000000000000 P b87cabdf7aa145babe0be5701e5c3815 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 21:26:34.795028  2093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b87cabdf7aa145babe0be5701e5c3815 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:34.795966  2093 raft_consensus.cc:686] T 00000000000000000000000000000000 P b87cabdf7aa145babe0be5701e5c3815 [term 1 LEADER]: Becoming Leader. State: Replica: b87cabdf7aa145babe0be5701e5c3815, State: Running, Role: LEADER
I20240502 21:26:34.796973  2093 consensus_queue.cc:237] T 00000000000000000000000000000000 P b87cabdf7aa145babe0be5701e5c3815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87cabdf7aa145babe0be5701e5c3815" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 43629 } }
I20240502 21:26:34.798322  2088 sys_catalog.cc:564] T 00000000000000000000000000000000 P b87cabdf7aa145babe0be5701e5c3815 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:26:34.801297  2095 sys_catalog.cc:455] T 00000000000000000000000000000000 P b87cabdf7aa145babe0be5701e5c3815 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b87cabdf7aa145babe0be5701e5c3815" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87cabdf7aa145babe0be5701e5c3815" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 43629 } } }
I20240502 21:26:34.801339  2094 sys_catalog.cc:455] T 00000000000000000000000000000000 P b87cabdf7aa145babe0be5701e5c3815 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b87cabdf7aa145babe0be5701e5c3815. Latest consensus state: current_term: 1 leader_uuid: "b87cabdf7aa145babe0be5701e5c3815" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87cabdf7aa145babe0be5701e5c3815" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 43629 } } }
I20240502 21:26:34.801715  2095 sys_catalog.cc:458] T 00000000000000000000000000000000 P b87cabdf7aa145babe0be5701e5c3815 [sys.catalog]: This master's current role is: LEADER
I20240502 21:26:34.801776  2094 sys_catalog.cc:458] T 00000000000000000000000000000000 P b87cabdf7aa145babe0be5701e5c3815 [sys.catalog]: This master's current role is: LEADER
I20240502 21:26:34.805950  2100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:26:34.813714  2100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:26:34.826728  2100 catalog_manager.cc:1347] Generated new cluster ID: 45ca5b8a088449938fa56d0bb4707f53
I20240502 21:26:34.826894  2100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:26:34.878211  2100 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:26:34.879745  2100 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:26:34.891534  2100 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b87cabdf7aa145babe0be5701e5c3815: Generated new TSK 0
I20240502 21:26:34.892714  2100 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:26:34.901895 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.65:0
--local_ip_for_outbound_sockets=127.27.196.65
--webserver_interface=127.27.196.65
--webserver_port=0
--tserver_master_addrs=127.27.196.126:43629
--builtin_ntp_servers=127.27.196.84:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 21:26:35.346985  2112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:35.347599  2112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:35.347772  2112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:35.357506  2112 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:35.357672  2112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:35.357851  2112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.65
I20240502 21:26:35.367933  2112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.27.196.65
--webserver_port=0
--tserver_master_addrs=127.27.196.126:43629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.65
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:35.369415  2112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:35.371393  2112 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:35.385923  2120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:35.386188  2117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:35.386644  2118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:35.387854  2112 server_base.cc:1034] running on GCE node
I20240502 21:26:35.388865  2112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:35.390340  2112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:35.391733  2112 hybrid_clock.cc:648] HybridClock initialized: now 1714685195391643 us; error 86 us; skew 500 ppm
I20240502 21:26:35.392159  2112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:35.396008  2112 webserver.cc:460] Webserver started at http://127.27.196.65:36051/ using document root <none> and password file <none>
I20240502 21:26:35.396953  2112 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:35.397086  2112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:35.397426  2112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:35.401435  2112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d961985de7874e1990a19bb9b2da14c2"
format_stamp: "Formatted at 2024-05-02 21:26:35 on dist-test-slave-3bs1"
I20240502 21:26:35.402590  2112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d961985de7874e1990a19bb9b2da14c2"
format_stamp: "Formatted at 2024-05-02 21:26:35 on dist-test-slave-3bs1"
I20240502 21:26:35.410743  2112 fs_manager.cc:691] Time spent creating directory manager: real 0.008s	user 0.010s	sys 0.000s
I20240502 21:26:35.416815  2126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:35.418897  2112 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.005s	sys 0.000s
I20240502 21:26:35.419193  2112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d961985de7874e1990a19bb9b2da14c2"
format_stamp: "Formatted at 2024-05-02 21:26:35 on dist-test-slave-3bs1"
I20240502 21:26:35.419456  2112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:35.439069  2112 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:26:35.440147  2112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:35.440461  2112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:35.441581  2112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:35.443571  2112 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:35.443681  2112 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:35.443807  2112 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:35.443881  2112 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:35.482448  2112 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.65:37025
I20240502 21:26:35.482507  2240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.65:37025 every 8 connection(s)
I20240502 21:26:35.483914  2112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:26:35.488049 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 2112
I20240502 21:26:35.488714 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:26:35.494421 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.66:0
--local_ip_for_outbound_sockets=127.27.196.66
--webserver_interface=127.27.196.66
--webserver_port=0
--tserver_master_addrs=127.27.196.126:43629
--builtin_ntp_servers=127.27.196.84:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:26:35.503490  2241 heartbeater.cc:347] Connected to a master server at 127.27.196.126:43629
I20240502 21:26:35.504092  2241 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:35.505385  2241 heartbeater.cc:511] Master 127.27.196.126:43629 requested a full tablet report, sending...
I20240502 21:26:35.508922  2052 ts_manager.cc:194] Registered new tserver with Master: d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025)
I20240502 21:26:35.511545  2052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.65:41373
W20240502 21:26:35.872845  2245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:35.873281  2245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:35.873387  2245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:35.882869  2245 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:35.883085  2245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:35.883308  2245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.66
I20240502 21:26:35.895227  2245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.27.196.66
--webserver_port=0
--tserver_master_addrs=127.27.196.126:43629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.66
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:35.897378  2245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:35.900206  2245 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:35.917411  2251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:35.917881  2252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:35.918773  2254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:35.919579  2245 server_base.cc:1034] running on GCE node
I20240502 21:26:35.920478  2245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:35.921687  2245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:35.922959  2245 hybrid_clock.cc:648] HybridClock initialized: now 1714685195922880 us; error 66 us; skew 500 ppm
I20240502 21:26:35.923416  2245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:35.926213  2245 webserver.cc:460] Webserver started at http://127.27.196.66:44911/ using document root <none> and password file <none>
I20240502 21:26:35.927079  2245 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:35.927304  2245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:35.927639  2245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:35.930580  2245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "68f426685b4c4c90bda61ef52177cc22"
format_stamp: "Formatted at 2024-05-02 21:26:35 on dist-test-slave-3bs1"
I20240502 21:26:35.931579  2245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "68f426685b4c4c90bda61ef52177cc22"
format_stamp: "Formatted at 2024-05-02 21:26:35 on dist-test-slave-3bs1"
I20240502 21:26:35.937914  2245 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.001s	sys 0.004s
I20240502 21:26:35.942600  2260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:35.944669  2245 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.002s
I20240502 21:26:35.944936  2245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "68f426685b4c4c90bda61ef52177cc22"
format_stamp: "Formatted at 2024-05-02 21:26:35 on dist-test-slave-3bs1"
I20240502 21:26:35.945225  2245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:35.976330  2245 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:26:35.977461  2245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:35.977813  2245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:35.978937  2245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:35.980764  2245 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:35.980882  2245 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:35.981017  2245 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:35.981097  2245 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:36.020356  2245 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.66:36253
I20240502 21:26:36.020501  2378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.66:36253 every 8 connection(s)
I20240502 21:26:36.021831  2245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:26:36.024084 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 2245
I20240502 21:26:36.024288 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:26:36.029315 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.67:0
--local_ip_for_outbound_sockets=127.27.196.67
--webserver_interface=127.27.196.67
--webserver_port=0
--tserver_master_addrs=127.27.196.126:43629
--builtin_ntp_servers=127.27.196.84:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:26:36.040701  2379 heartbeater.cc:347] Connected to a master server at 127.27.196.126:43629
I20240502 21:26:36.041102  2379 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:36.042452  2379 heartbeater.cc:511] Master 127.27.196.126:43629 requested a full tablet report, sending...
I20240502 21:26:36.044858  2052 ts_manager.cc:194] Registered new tserver with Master: 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253)
I20240502 21:26:36.045940  2052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.66:60537
W20240502 21:26:36.454957  2383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:36.455461  2383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:36.455623  2383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:36.464438  2383 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:36.464679  2383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:36.464881  2383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.67
I20240502 21:26:36.476013  2383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.27.196.67
--webserver_port=0
--tserver_master_addrs=127.27.196.126:43629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.67
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:36.478076  2383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:36.480651  2383 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:36.499446  2388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:36.499928  2389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:36.500465  2391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:36.500936  2383 server_base.cc:1034] running on GCE node
I20240502 21:26:36.501686  2383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:36.502985  2383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:36.504595  2383 hybrid_clock.cc:648] HybridClock initialized: now 1714685196504534 us; error 68 us; skew 500 ppm
I20240502 21:26:36.504976  2383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:36.507849  2383 webserver.cc:460] Webserver started at http://127.27.196.67:42753/ using document root <none> and password file <none>
I20240502 21:26:36.508754  2383 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:36.508867  2383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:36.509234  2383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:36.512300  2383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "c616208720db4b14a9219cae8f2b7d49"
format_stamp: "Formatted at 2024-05-02 21:26:36 on dist-test-slave-3bs1"
I20240502 21:26:36.513200  2383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "c616208720db4b14a9219cae8f2b7d49"
format_stamp: "Formatted at 2024-05-02 21:26:36 on dist-test-slave-3bs1"
I20240502 21:26:36.515377  2241 heartbeater.cc:503] Master 127.27.196.126:43629 was elected leader, sending a full tablet report...
I20240502 21:26:36.519794  2383 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.007s	sys 0.000s
I20240502 21:26:36.525198  2397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:36.527058  2383 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.005s	sys 0.000s
I20240502 21:26:36.527313  2383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c616208720db4b14a9219cae8f2b7d49"
format_stamp: "Formatted at 2024-05-02 21:26:36 on dist-test-slave-3bs1"
I20240502 21:26:36.527544  2383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:36.549854  2383 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:26:36.550998  2383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:36.551424  2383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:36.552690  2383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:36.554447  2383 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:36.554555  2383 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:36.554685  2383 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:36.554754  2383 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:36.595049  2383 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.67:46687
I20240502 21:26:36.595140  2512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.67:46687 every 8 connection(s)
I20240502 21:26:36.596606  2383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:26:36.602376 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 2383
I20240502 21:26:36.602694 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:26:36.607197 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.68:0
--local_ip_for_outbound_sockets=127.27.196.68
--webserver_interface=127.27.196.68
--webserver_port=0
--tserver_master_addrs=127.27.196.126:43629
--builtin_ntp_servers=127.27.196.84:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:26:36.618232  2513 heartbeater.cc:347] Connected to a master server at 127.27.196.126:43629
I20240502 21:26:36.618640  2513 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:36.619607  2513 heartbeater.cc:511] Master 127.27.196.126:43629 requested a full tablet report, sending...
I20240502 21:26:36.621495  2052 ts_manager.cc:194] Registered new tserver with Master: c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687)
I20240502 21:26:36.622376  2052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.67:52869
W20240502 21:26:36.988569  2517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:36.988934  2517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:36.989090  2517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:36.998050  2517 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:36.998261  2517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:36.998417  2517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.68
I20240502 21:26:37.008699  2517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.27.196.68
--webserver_port=0
--tserver_master_addrs=127.27.196.126:43629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.68
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:37.010382  2517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:37.012554  2517 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:37.031948  2523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:37.032670  2524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:37.033113  2526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:37.034209  2517 server_base.cc:1034] running on GCE node
I20240502 21:26:37.035283  2517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:37.037133  2517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:37.038463  2517 hybrid_clock.cc:648] HybridClock initialized: now 1714685197038402 us; error 66 us; skew 500 ppm
I20240502 21:26:37.038955  2517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:37.042234  2517 webserver.cc:460] Webserver started at http://127.27.196.68:40277/ using document root <none> and password file <none>
I20240502 21:26:37.043479  2517 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:37.043663  2517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:37.044107  2517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:37.047992  2517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "13451be0f7134a2c89c0c9330177213e"
format_stamp: "Formatted at 2024-05-02 21:26:37 on dist-test-slave-3bs1"
I20240502 21:26:37.049453  2517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "13451be0f7134a2c89c0c9330177213e"
format_stamp: "Formatted at 2024-05-02 21:26:37 on dist-test-slave-3bs1"
I20240502 21:26:37.049916  2379 heartbeater.cc:503] Master 127.27.196.126:43629 was elected leader, sending a full tablet report...
I20240502 21:26:37.057874  2517 fs_manager.cc:691] Time spent creating directory manager: real 0.008s	user 0.009s	sys 0.000s
I20240502 21:26:37.063000  2532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:37.065101  2517 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 21:26:37.065349  2517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "13451be0f7134a2c89c0c9330177213e"
format_stamp: "Formatted at 2024-05-02 21:26:37 on dist-test-slave-3bs1"
I20240502 21:26:37.065609  2517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:37.119233  2517 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:26:37.120214  2517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:37.120488  2517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:37.121647  2517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:37.123502  2517 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:37.123605  2517 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:37.123699  2517 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:37.123757  2517 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:37.163444  2517 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.68:44575
I20240502 21:26:37.163535  2645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.68:44575 every 8 connection(s)
I20240502 21:26:37.164928  2517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:26:37.168378 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 2517
I20240502 21:26:37.168702 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 21:26:37.173431 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.69:0
--local_ip_for_outbound_sockets=127.27.196.69
--webserver_interface=127.27.196.69
--webserver_port=0
--tserver_master_addrs=127.27.196.126:43629
--builtin_ntp_servers=127.27.196.84:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:26:37.182825  2646 heartbeater.cc:347] Connected to a master server at 127.27.196.126:43629
I20240502 21:26:37.183332  2646 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:37.184548  2646 heartbeater.cc:511] Master 127.27.196.126:43629 requested a full tablet report, sending...
I20240502 21:26:37.186898  2052 ts_manager.cc:194] Registered new tserver with Master: 13451be0f7134a2c89c0c9330177213e (127.27.196.68:44575)
I20240502 21:26:37.188611  2052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.68:53327
W20240502 21:26:37.590400  2650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:37.590885  2650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:37.591040  2650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:37.602371  2650 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:37.602530  2650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:37.602658  2650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.69
I20240502 21:26:37.613121  2650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.27.196.69
--webserver_port=0
--tserver_master_addrs=127.27.196.126:43629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.69
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:37.614823  2650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:37.616962  2650 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 21:26:37.625897  2513 heartbeater.cc:503] Master 127.27.196.126:43629 was elected leader, sending a full tablet report...
W20240502 21:26:37.631430  2657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:37.631987  2655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:37.632787  2659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:37.633170  2650 server_base.cc:1034] running on GCE node
I20240502 21:26:37.633883  2650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:37.634981  2650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:37.636222  2650 hybrid_clock.cc:648] HybridClock initialized: now 1714685197636173 us; error 47 us; skew 500 ppm
I20240502 21:26:37.636602  2650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:37.639341  2650 webserver.cc:460] Webserver started at http://127.27.196.69:38529/ using document root <none> and password file <none>
I20240502 21:26:37.640195  2650 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:37.640324  2650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:37.640688  2650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:37.643687  2650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "4e8c16d1a1484a41a54b5ddc198becd3"
format_stamp: "Formatted at 2024-05-02 21:26:37 on dist-test-slave-3bs1"
I20240502 21:26:37.644554  2650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "4e8c16d1a1484a41a54b5ddc198becd3"
format_stamp: "Formatted at 2024-05-02 21:26:37 on dist-test-slave-3bs1"
I20240502 21:26:37.651423  2650 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.001s
I20240502 21:26:37.655655  2665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:37.657353  2650 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
I20240502 21:26:37.657581  2650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "4e8c16d1a1484a41a54b5ddc198becd3"
format_stamp: "Formatted at 2024-05-02 21:26:37 on dist-test-slave-3bs1"
I20240502 21:26:37.657905  2650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:37.679831  2650 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:26:37.680861  2650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:37.681183  2650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:37.682459  2650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:37.684365  2650 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:37.684486  2650 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:37.684623  2650 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:37.684700  2650 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:37.725967  2650 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.69:37783
I20240502 21:26:37.726094  2779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.69:37783 every 8 connection(s)
I20240502 21:26:37.727656  2650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:26:37.735474 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 2650
I20240502 21:26:37.735702 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 21:26:37.739099 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.70:0
--local_ip_for_outbound_sockets=127.27.196.70
--webserver_interface=127.27.196.70
--webserver_port=0
--tserver_master_addrs=127.27.196.126:43629
--builtin_ntp_servers=127.27.196.84:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:26:37.744277  2780 heartbeater.cc:347] Connected to a master server at 127.27.196.126:43629
I20240502 21:26:37.744666  2780 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:37.745501  2780 heartbeater.cc:511] Master 127.27.196.126:43629 requested a full tablet report, sending...
I20240502 21:26:37.747638  2052 ts_manager.cc:194] Registered new tserver with Master: 4e8c16d1a1484a41a54b5ddc198becd3 (127.27.196.69:37783)
I20240502 21:26:37.748881  2052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.69:42291
W20240502 21:26:38.120690  2784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:38.121040  2784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:38.121140  2784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:38.129534  2784 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:38.129691  2784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:38.129829  2784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.70
I20240502 21:26:38.141077  2784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.27.196.70
--webserver_port=0
--tserver_master_addrs=127.27.196.126:43629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.70
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:38.143024  2784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:38.145524  2784 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:38.166550  2789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:38.166919  2790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:38.166389  2792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:38.169582  2784 server_base.cc:1034] running on GCE node
I20240502 21:26:38.170516  2784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:38.171998  2784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:38.175380  2784 hybrid_clock.cc:648] HybridClock initialized: now 1714685198175304 us; error 77 us; skew 500 ppm
I20240502 21:26:38.175832  2784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:38.183874  2784 webserver.cc:460] Webserver started at http://127.27.196.70:36905/ using document root <none> and password file <none>
I20240502 21:26:38.184815  2784 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:38.184967  2784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:38.185446  2784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:38.188411  2784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "1193f1d0d2c843139d98458cf4035eb8"
format_stamp: "Formatted at 2024-05-02 21:26:38 on dist-test-slave-3bs1"
I20240502 21:26:38.189287  2784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "1193f1d0d2c843139d98458cf4035eb8"
format_stamp: "Formatted at 2024-05-02 21:26:38 on dist-test-slave-3bs1"
I20240502 21:26:38.192667  2646 heartbeater.cc:503] Master 127.27.196.126:43629 was elected leader, sending a full tablet report...
I20240502 21:26:38.196393  2784 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.007s	sys 0.001s
I20240502 21:26:38.203532  2798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:38.206204  2784 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.004s	sys 0.000s
I20240502 21:26:38.206485  2784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "1193f1d0d2c843139d98458cf4035eb8"
format_stamp: "Formatted at 2024-05-02 21:26:38 on dist-test-slave-3bs1"
I20240502 21:26:38.206930  2784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:38.235612  2784 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:26:38.236930  2784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:38.237339  2784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:38.238772  2784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:38.241693  2784 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:38.241906  2784 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:38.242076  2784 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:38.242198  2784 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:38.284103  2784 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.70:46179
I20240502 21:26:38.284163  2911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.70:46179 every 8 connection(s)
I20240502 21:26:38.285668  2784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 21:26:38.287606 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 2784
I20240502 21:26:38.287796 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 21:26:38.291070 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.71:0
--local_ip_for_outbound_sockets=127.27.196.71
--webserver_interface=127.27.196.71
--webserver_port=0
--tserver_master_addrs=127.27.196.126:43629
--builtin_ntp_servers=127.27.196.84:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:26:38.307325  2912 heartbeater.cc:347] Connected to a master server at 127.27.196.126:43629
I20240502 21:26:38.307799  2912 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:38.309124  2912 heartbeater.cc:511] Master 127.27.196.126:43629 requested a full tablet report, sending...
I20240502 21:26:38.311630  2052 ts_manager.cc:194] Registered new tserver with Master: 1193f1d0d2c843139d98458cf4035eb8 (127.27.196.70:46179)
I20240502 21:26:38.312583  2052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.70:38591
W20240502 21:26:38.690672  2916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:38.691224  2916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:38.691427  2916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:38.700700  2916 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:38.700892  2916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:38.701081  2916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.71
I20240502 21:26:38.713338  2916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.27.196.71
--webserver_port=0
--tserver_master_addrs=127.27.196.126:43629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.71
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:38.715412  2916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:38.717833  2916 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:38.736764  2921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:38.737355  2922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:38.738090  2924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:38.738878  2916 server_base.cc:1034] running on GCE node
I20240502 21:26:38.739672  2916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:38.740971  2916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:38.742398  2916 hybrid_clock.cc:648] HybridClock initialized: now 1714685198742317 us; error 77 us; skew 500 ppm
I20240502 21:26:38.742892  2916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:38.746160  2916 webserver.cc:460] Webserver started at http://127.27.196.71:43309/ using document root <none> and password file <none>
I20240502 21:26:38.747095  2916 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:38.747288  2916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:38.747668  2916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:38.750806  2916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "d306b26127e04e8c85a98edf1b41978a"
format_stamp: "Formatted at 2024-05-02 21:26:38 on dist-test-slave-3bs1"
I20240502 21:26:38.751942  2916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "d306b26127e04e8c85a98edf1b41978a"
format_stamp: "Formatted at 2024-05-02 21:26:38 on dist-test-slave-3bs1"
I20240502 21:26:38.752914  2780 heartbeater.cc:503] Master 127.27.196.126:43629 was elected leader, sending a full tablet report...
I20240502 21:26:38.759321  2916 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.007s	sys 0.000s
I20240502 21:26:38.764568  2930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:38.766544  2916 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
I20240502 21:26:38.766816  2916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "d306b26127e04e8c85a98edf1b41978a"
format_stamp: "Formatted at 2024-05-02 21:26:38 on dist-test-slave-3bs1"
I20240502 21:26:38.767082  2916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:38.794633  2916 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:26:38.795708  2916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:38.796041  2916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:38.797190  2916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:38.799167  2916 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:38.799281  2916 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:38.799419  2916 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:38.799521  2916 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:38.838768  2916 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.71:39583
I20240502 21:26:38.838814  3044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.71:39583 every 8 connection(s)
I20240502 21:26:38.840324  2916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 21:26:38.850343 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 2916
I20240502 21:26:38.850610 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 21:26:38.858981  3045 heartbeater.cc:347] Connected to a master server at 127.27.196.126:43629
I20240502 21:26:38.859493  3045 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:38.860440  3045 heartbeater.cc:511] Master 127.27.196.126:43629 requested a full tablet report, sending...
I20240502 21:26:38.862699  2052 ts_manager.cc:194] Registered new tserver with Master: d306b26127e04e8c85a98edf1b41978a (127.27.196.71:39583)
I20240502 21:26:38.863794  2052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.71:51681
I20240502 21:26:38.870374 28433 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 21:26:38.891903  2052 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52214:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240502 21:26:38.940495  2581 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae512d4d5874a909b1f4186c4b9785a (DEFAULT_TABLE table=TestTable [id=4c82bfdabd1144f59d08b3d9f3d5cb56]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:38.942180  2581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae512d4d5874a909b1f4186c4b9785a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:38.941718  2309 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae512d4d5874a909b1f4186c4b9785a (DEFAULT_TABLE table=TestTable [id=4c82bfdabd1144f59d08b3d9f3d5cb56]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:38.943171  2309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae512d4d5874a909b1f4186c4b9785a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:38.954231  3062 tablet_bootstrap.cc:492] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22: Bootstrap starting.
I20240502 21:26:38.954741  3063 tablet_bootstrap.cc:492] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e: Bootstrap starting.
I20240502 21:26:38.957815  3063 tablet_bootstrap.cc:654] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:38.957818  3062 tablet_bootstrap.cc:654] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:38.959030  3063 log.cc:824] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:38.959061  3062 log.cc:824] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:38.962628  3063 tablet_bootstrap.cc:492] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e: No bootstrap required, opened a new log
I20240502 21:26:38.962631  3062 tablet_bootstrap.cc:492] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22: No bootstrap required, opened a new log
I20240502 21:26:38.963022  3062 ts_tablet_manager.cc:1367] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22: Time spent bootstrapping tablet: real 0.009s	user 0.006s	sys 0.000s
I20240502 21:26:38.963003  3063 ts_tablet_manager.cc:1367] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e: Time spent bootstrapping tablet: real 0.009s	user 0.005s	sys 0.000s
I20240502 21:26:38.973050  3063 raft_consensus.cc:348] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:38.973052  3062 raft_consensus.cc:348] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:38.973611  3062 raft_consensus.cc:374] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:38.973611  3063 raft_consensus.cc:374] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:38.973734  3063 raft_consensus.cc:729] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13451be0f7134a2c89c0c9330177213e, State: Initialized, Role: FOLLOWER
I20240502 21:26:38.973734  3062 raft_consensus.cc:729] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68f426685b4c4c90bda61ef52177cc22, State: Initialized, Role: FOLLOWER
I20240502 21:26:38.974577  3063 consensus_queue.cc:260] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:38.974577  3062 consensus_queue.cc:260] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:38.976910  3063 ts_tablet_manager.cc:1398] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.004s
I20240502 21:26:38.979449  3062 ts_tablet_manager.cc:1398] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22: Time spent starting tablet: real 0.016s	user 0.010s	sys 0.004s
W20240502 21:26:39.026780  2380 tablet.cc:2369] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:39.074030  3066 raft_consensus.cc:482] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:39.074316  3066 raft_consensus.cc:504] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:39.076340  3066 leader_election.cc:288] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253)
I20240502 21:26:39.083611  2330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae512d4d5874a909b1f4186c4b9785a" candidate_uuid: "13451be0f7134a2c89c0c9330177213e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68f426685b4c4c90bda61ef52177cc22" is_pre_election: true
I20240502 21:26:39.084175  2330 raft_consensus.cc:2454] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13451be0f7134a2c89c0c9330177213e in term 0.
I20240502 21:26:39.085472  2533 leader_election.cc:302] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 13451be0f7134a2c89c0c9330177213e, 68f426685b4c4c90bda61ef52177cc22; no voters: 
I20240502 21:26:39.086148  3066 raft_consensus.cc:2781] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:39.086369  3066 raft_consensus.cc:482] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:39.086696  3066 raft_consensus.cc:3037] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:39.089476  3066 raft_consensus.cc:504] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:39.090109  3066 leader_election.cc:288] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [CANDIDATE]: Term 1 election: Requested vote from peers 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253)
I20240502 21:26:39.090975  2330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae512d4d5874a909b1f4186c4b9785a" candidate_uuid: "13451be0f7134a2c89c0c9330177213e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68f426685b4c4c90bda61ef52177cc22"
I20240502 21:26:39.091218  2330 raft_consensus.cc:3037] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:39.093541  2330 raft_consensus.cc:2454] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13451be0f7134a2c89c0c9330177213e in term 1.
I20240502 21:26:39.094098  2533 leader_election.cc:302] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 13451be0f7134a2c89c0c9330177213e, 68f426685b4c4c90bda61ef52177cc22; no voters: 
I20240502 21:26:39.094736  3066 raft_consensus.cc:2781] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:39.095316  3066 raft_consensus.cc:686] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [term 1 LEADER]: Becoming Leader. State: Replica: 13451be0f7134a2c89c0c9330177213e, State: Running, Role: LEADER
I20240502 21:26:39.095939  3066 consensus_queue.cc:237] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:39.101050  2052 catalog_manager.cc:5484] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e reported cstate change: term changed from 0 to 1, leader changed from <none> to 13451be0f7134a2c89c0c9330177213e (127.27.196.68). New cstate: current_term: 1 leader_uuid: "13451be0f7134a2c89c0c9330177213e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } }
I20240502 21:26:39.121716 28433 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 21:26:39.124838 28433 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d961985de7874e1990a19bb9b2da14c2 to finish bootstrapping
I20240502 21:26:39.136123 28433 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 68f426685b4c4c90bda61ef52177cc22 to finish bootstrapping
I20240502 21:26:39.146518 28433 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c616208720db4b14a9219cae8f2b7d49 to finish bootstrapping
I20240502 21:26:39.158223 28433 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 13451be0f7134a2c89c0c9330177213e to finish bootstrapping
W20240502 21:26:39.168555  2647 tablet.cc:2369] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:39.169867 28433 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4e8c16d1a1484a41a54b5ddc198becd3 to finish bootstrapping
I20240502 21:26:39.181403 28433 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 1193f1d0d2c843139d98458cf4035eb8 to finish bootstrapping
I20240502 21:26:39.192158 28433 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d306b26127e04e8c85a98edf1b41978a to finish bootstrapping
I20240502 21:26:39.202775 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 2517
I20240502 21:26:39.204159  2641 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:39.302100 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 2517
I20240502 21:26:39.316118  2912 heartbeater.cc:503] Master 127.27.196.126:43629 was elected leader, sending a full tablet report...
I20240502 21:26:39.323681 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 2650
I20240502 21:26:39.324939  2775 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:39.411746 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 2650
I20240502 21:26:39.431423 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 2784
I20240502 21:26:39.432785  2907 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:39.528029 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 2784
I20240502 21:26:39.553207 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 2916
I20240502 21:26:39.554980  3040 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:39.645551 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 2916
I20240502 21:26:40.898056  3081 raft_consensus.cc:482] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:40.898296  3081 raft_consensus.cc:504] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:40.900746  3081 leader_election.cc:288] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13451be0f7134a2c89c0c9330177213e (127.27.196.68:44575)
W20240502 21:26:40.902284  2265 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.68:44575: connect: Connection refused (error 111)
W20240502 21:26:40.904662  2265 leader_election.cc:334] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 13451be0f7134a2c89c0c9330177213e (127.27.196.68:44575): Network error: Client connection negotiation failed: client connection to 127.27.196.68:44575: connect: Connection refused (error 111)
I20240502 21:26:40.904968  2265 leader_election.cc:302] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 68f426685b4c4c90bda61ef52177cc22; no voters: 13451be0f7134a2c89c0c9330177213e
I20240502 21:26:40.905689  3081 raft_consensus.cc:2726] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:26:42.760600  3085 raft_consensus.cc:482] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:42.760804  3085 raft_consensus.cc:504] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:42.761444  3085 leader_election.cc:288] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13451be0f7134a2c89c0c9330177213e (127.27.196.68:44575)
W20240502 21:26:42.764366  2265 leader_election.cc:334] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 13451be0f7134a2c89c0c9330177213e (127.27.196.68:44575): Network error: Client connection negotiation failed: client connection to 127.27.196.68:44575: connect: Connection refused (error 111)
I20240502 21:26:42.764511  2265 leader_election.cc:302] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 68f426685b4c4c90bda61ef52177cc22; no voters: 13451be0f7134a2c89c0c9330177213e
I20240502 21:26:42.764899  3085 raft_consensus.cc:2726] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:26:43.419873  2052 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52214:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 21:26:43.438234  2309 tablet_service.cc:1467] Processing CreateTablet for tablet d7db9b73c0594885b383b059520414e9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d7b10d11f7af4b32bf7fda57d6b60382]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:43.438845  2309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7db9b73c0594885b383b059520414e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:43.448745  3097 tablet_bootstrap.cc:492] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22: Bootstrap starting.
I20240502 21:26:43.453236  3097 tablet_bootstrap.cc:654] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:43.453450  2174 tablet_service.cc:1467] Processing CreateTablet for tablet fe578b007ccd496ba2f70cda8e90c91f (DEFAULT_TABLE table=rebalance_test_table_0 [id=d7b10d11f7af4b32bf7fda57d6b60382]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:43.453938  2446 tablet_service.cc:1467] Processing CreateTablet for tablet fe578b007ccd496ba2f70cda8e90c91f (DEFAULT_TABLE table=rebalance_test_table_0 [id=d7b10d11f7af4b32bf7fda57d6b60382]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:43.455615  2446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe578b007ccd496ba2f70cda8e90c91f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:43.455631  2174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe578b007ccd496ba2f70cda8e90c91f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:43.455497  2173 tablet_service.cc:1467] Processing CreateTablet for tablet d7db9b73c0594885b383b059520414e9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d7b10d11f7af4b32bf7fda57d6b60382]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:43.456604  2173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7db9b73c0594885b383b059520414e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:43.455799  2447 tablet_service.cc:1467] Processing CreateTablet for tablet 6e1c5a5cc57243d7bf6e212e3a32dfe3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d7b10d11f7af4b32bf7fda57d6b60382]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:43.457988  2447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e1c5a5cc57243d7bf6e212e3a32dfe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:43.453557  2175 tablet_service.cc:1467] Processing CreateTablet for tablet 6e1c5a5cc57243d7bf6e212e3a32dfe3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d7b10d11f7af4b32bf7fda57d6b60382]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:43.467610  2175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e1c5a5cc57243d7bf6e212e3a32dfe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:43.476780  3097 tablet_bootstrap.cc:492] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22: No bootstrap required, opened a new log
I20240502 21:26:43.477152  3097 ts_tablet_manager.cc:1367] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22: Time spent bootstrapping tablet: real 0.029s	user 0.015s	sys 0.011s
I20240502 21:26:43.478726  3097 raft_consensus.cc:348] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:43.479197  3097 raft_consensus.cc:374] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:43.479354  3097 raft_consensus.cc:729] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68f426685b4c4c90bda61ef52177cc22, State: Initialized, Role: FOLLOWER
I20240502 21:26:43.479682  3097 consensus_queue.cc:260] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:43.484869  3099 tablet_bootstrap.cc:492] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P c616208720db4b14a9219cae8f2b7d49: Bootstrap starting.
I20240502 21:26:43.488801  3097 ts_tablet_manager.cc:1398] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22: Time spent starting tablet: real 0.011s	user 0.002s	sys 0.006s
I20240502 21:26:43.489125  3099 tablet_bootstrap.cc:654] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P c616208720db4b14a9219cae8f2b7d49: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:43.490643  3099 log.cc:824] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P c616208720db4b14a9219cae8f2b7d49: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:43.493224  3101 tablet_bootstrap.cc:492] T d7db9b73c0594885b383b059520414e9 P d961985de7874e1990a19bb9b2da14c2: Bootstrap starting.
I20240502 21:26:43.495867  3099 tablet_bootstrap.cc:492] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P c616208720db4b14a9219cae8f2b7d49: No bootstrap required, opened a new log
I20240502 21:26:43.496261  3099 ts_tablet_manager.cc:1367] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P c616208720db4b14a9219cae8f2b7d49: Time spent bootstrapping tablet: real 0.012s	user 0.009s	sys 0.000s
I20240502 21:26:43.497467  3101 tablet_bootstrap.cc:654] T d7db9b73c0594885b383b059520414e9 P d961985de7874e1990a19bb9b2da14c2: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:43.498976  3101 log.cc:824] T d7db9b73c0594885b383b059520414e9 P d961985de7874e1990a19bb9b2da14c2: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:43.503486  3101 tablet_bootstrap.cc:492] T d7db9b73c0594885b383b059520414e9 P d961985de7874e1990a19bb9b2da14c2: No bootstrap required, opened a new log
I20240502 21:26:43.503849  3101 ts_tablet_manager.cc:1367] T d7db9b73c0594885b383b059520414e9 P d961985de7874e1990a19bb9b2da14c2: Time spent bootstrapping tablet: real 0.011s	user 0.006s	sys 0.004s
I20240502 21:26:43.505928  3099 raft_consensus.cc:348] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:43.506444  3099 raft_consensus.cc:374] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:43.506552  3099 raft_consensus.cc:729] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c616208720db4b14a9219cae8f2b7d49, State: Initialized, Role: FOLLOWER
I20240502 21:26:43.507494  3099 consensus_queue.cc:260] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P c616208720db4b14a9219cae8f2b7d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:43.510743  3099 ts_tablet_manager.cc:1398] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P c616208720db4b14a9219cae8f2b7d49: Time spent starting tablet: real 0.014s	user 0.013s	sys 0.000s
I20240502 21:26:43.513008  3099 tablet_bootstrap.cc:492] T fe578b007ccd496ba2f70cda8e90c91f P c616208720db4b14a9219cae8f2b7d49: Bootstrap starting.
I20240502 21:26:43.512813  3101 raft_consensus.cc:348] T d7db9b73c0594885b383b059520414e9 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:43.513353  3101 raft_consensus.cc:374] T d7db9b73c0594885b383b059520414e9 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:43.513489  3101 raft_consensus.cc:729] T d7db9b73c0594885b383b059520414e9 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d961985de7874e1990a19bb9b2da14c2, State: Initialized, Role: FOLLOWER
I20240502 21:26:43.514288  3101 consensus_queue.cc:260] T d7db9b73c0594885b383b059520414e9 P d961985de7874e1990a19bb9b2da14c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:43.516501  3101 ts_tablet_manager.cc:1398] T d7db9b73c0594885b383b059520414e9 P d961985de7874e1990a19bb9b2da14c2: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.005s
I20240502 21:26:43.516600  3099 tablet_bootstrap.cc:654] T fe578b007ccd496ba2f70cda8e90c91f P c616208720db4b14a9219cae8f2b7d49: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:43.516990  3101 tablet_bootstrap.cc:492] T fe578b007ccd496ba2f70cda8e90c91f P d961985de7874e1990a19bb9b2da14c2: Bootstrap starting.
I20240502 21:26:43.518839  3101 tablet_bootstrap.cc:654] T fe578b007ccd496ba2f70cda8e90c91f P d961985de7874e1990a19bb9b2da14c2: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:43.521191  3101 tablet_bootstrap.cc:492] T fe578b007ccd496ba2f70cda8e90c91f P d961985de7874e1990a19bb9b2da14c2: No bootstrap required, opened a new log
I20240502 21:26:43.521359  3101 ts_tablet_manager.cc:1367] T fe578b007ccd496ba2f70cda8e90c91f P d961985de7874e1990a19bb9b2da14c2: Time spent bootstrapping tablet: real 0.004s	user 0.001s	sys 0.002s
I20240502 21:26:43.522711  3101 raft_consensus.cc:348] T fe578b007ccd496ba2f70cda8e90c91f P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:43.523056  3099 tablet_bootstrap.cc:492] T fe578b007ccd496ba2f70cda8e90c91f P c616208720db4b14a9219cae8f2b7d49: No bootstrap required, opened a new log
I20240502 21:26:43.523049  3101 raft_consensus.cc:374] T fe578b007ccd496ba2f70cda8e90c91f P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:43.523200  3101 raft_consensus.cc:729] T fe578b007ccd496ba2f70cda8e90c91f P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d961985de7874e1990a19bb9b2da14c2, State: Initialized, Role: FOLLOWER
I20240502 21:26:43.523308  3099 ts_tablet_manager.cc:1367] T fe578b007ccd496ba2f70cda8e90c91f P c616208720db4b14a9219cae8f2b7d49: Time spent bootstrapping tablet: real 0.010s	user 0.006s	sys 0.001s
I20240502 21:26:43.523583  3101 consensus_queue.cc:260] T fe578b007ccd496ba2f70cda8e90c91f P d961985de7874e1990a19bb9b2da14c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:43.524256  3101 ts_tablet_manager.cc:1398] T fe578b007ccd496ba2f70cda8e90c91f P d961985de7874e1990a19bb9b2da14c2: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 21:26:43.524633  3101 tablet_bootstrap.cc:492] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2: Bootstrap starting.
I20240502 21:26:43.524605  3099 raft_consensus.cc:348] T fe578b007ccd496ba2f70cda8e90c91f P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:43.524838  3099 raft_consensus.cc:374] T fe578b007ccd496ba2f70cda8e90c91f P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:43.524942  3099 raft_consensus.cc:729] T fe578b007ccd496ba2f70cda8e90c91f P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c616208720db4b14a9219cae8f2b7d49, State: Initialized, Role: FOLLOWER
I20240502 21:26:43.525322  3099 consensus_queue.cc:260] T fe578b007ccd496ba2f70cda8e90c91f P c616208720db4b14a9219cae8f2b7d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:43.525933  3099 ts_tablet_manager.cc:1398] T fe578b007ccd496ba2f70cda8e90c91f P c616208720db4b14a9219cae8f2b7d49: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 21:26:43.527079  3101 tablet_bootstrap.cc:654] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:43.530037  3101 tablet_bootstrap.cc:492] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2: No bootstrap required, opened a new log
I20240502 21:26:43.530295  3101 ts_tablet_manager.cc:1367] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240502 21:26:43.531096  3101 raft_consensus.cc:348] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:43.531420  3101 raft_consensus.cc:374] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:43.531508  3101 raft_consensus.cc:729] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d961985de7874e1990a19bb9b2da14c2, State: Initialized, Role: FOLLOWER
I20240502 21:26:43.531769  3101 consensus_queue.cc:260] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:43.532269  3101 ts_tablet_manager.cc:1398] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:26:43.553974  3100 raft_consensus.cc:482] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:43.554214  3100 raft_consensus.cc:504] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:43.555447  3105 raft_consensus.cc:482] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:43.555630  3100 leader_election.cc:288] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025)
I20240502 21:26:43.555742  3105 raft_consensus.cc:504] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:43.558045  3105 leader_election.cc:288] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687)
I20240502 21:26:43.565389  2195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7db9b73c0594885b383b059520414e9" candidate_uuid: "68f426685b4c4c90bda61ef52177cc22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d961985de7874e1990a19bb9b2da14c2" is_pre_election: true
I20240502 21:26:43.566174  2195 raft_consensus.cc:2454] T d7db9b73c0594885b383b059520414e9 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68f426685b4c4c90bda61ef52177cc22 in term 0.
I20240502 21:26:43.567279  2264 leader_election.cc:302] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 68f426685b4c4c90bda61ef52177cc22, d961985de7874e1990a19bb9b2da14c2; no voters: 
I20240502 21:26:43.567855  3100 raft_consensus.cc:2781] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:43.568045  3100 raft_consensus.cc:482] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:43.567690  2467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e1c5a5cc57243d7bf6e212e3a32dfe3" candidate_uuid: "d961985de7874e1990a19bb9b2da14c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c616208720db4b14a9219cae8f2b7d49" is_pre_election: true
I20240502 21:26:43.568161  3100 raft_consensus.cc:3037] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:43.568578  2467 raft_consensus.cc:2454] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d961985de7874e1990a19bb9b2da14c2 in term 0.
I20240502 21:26:43.569490  2128 leader_election.cc:302] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c616208720db4b14a9219cae8f2b7d49, d961985de7874e1990a19bb9b2da14c2; no voters: 
I20240502 21:26:43.570086  3105 raft_consensus.cc:2781] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:43.570354  3105 raft_consensus.cc:482] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:43.570477  3105 raft_consensus.cc:3037] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:43.571913  3100 raft_consensus.cc:504] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:43.572621  3100 leader_election.cc:288] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 1 election: Requested vote from peers d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025)
I20240502 21:26:43.573370  3105 raft_consensus.cc:504] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:43.573508  2195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7db9b73c0594885b383b059520414e9" candidate_uuid: "68f426685b4c4c90bda61ef52177cc22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d961985de7874e1990a19bb9b2da14c2"
I20240502 21:26:43.573796  2195 raft_consensus.cc:3037] T d7db9b73c0594885b383b059520414e9 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:43.574451  3105 leader_election.cc:288] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [CANDIDATE]: Term 1 election: Requested vote from peers c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687)
I20240502 21:26:43.575237  2467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e1c5a5cc57243d7bf6e212e3a32dfe3" candidate_uuid: "d961985de7874e1990a19bb9b2da14c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c616208720db4b14a9219cae8f2b7d49"
I20240502 21:26:43.575496  2467 raft_consensus.cc:3037] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:43.576810  2195 raft_consensus.cc:2454] T d7db9b73c0594885b383b059520414e9 P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68f426685b4c4c90bda61ef52177cc22 in term 1.
I20240502 21:26:43.577421  2264 leader_election.cc:302] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 68f426685b4c4c90bda61ef52177cc22, d961985de7874e1990a19bb9b2da14c2; no voters: 
I20240502 21:26:43.577802  3100 raft_consensus.cc:2781] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:43.578291  2467 raft_consensus.cc:2454] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d961985de7874e1990a19bb9b2da14c2 in term 1.
I20240502 21:26:43.578424  3100 raft_consensus.cc:686] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Becoming Leader. State: Replica: 68f426685b4c4c90bda61ef52177cc22, State: Running, Role: LEADER
I20240502 21:26:43.578845  2128 leader_election.cc:302] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c616208720db4b14a9219cae8f2b7d49, d961985de7874e1990a19bb9b2da14c2; no voters: 
I20240502 21:26:43.579319  3105 raft_consensus.cc:2781] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:43.579418  3100 consensus_queue.cc:237] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:43.579850  3105 raft_consensus.cc:686] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [term 1 LEADER]: Becoming Leader. State: Replica: d961985de7874e1990a19bb9b2da14c2, State: Running, Role: LEADER
I20240502 21:26:43.580381  3105 consensus_queue.cc:237] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:43.586256  2052 catalog_manager.cc:5484] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d961985de7874e1990a19bb9b2da14c2 (127.27.196.65). New cstate: current_term: 1 leader_uuid: "d961985de7874e1990a19bb9b2da14c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } }
I20240502 21:26:43.586719  2051 catalog_manager.cc:5484] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "68f426685b4c4c90bda61ef52177cc22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } }
W20240502 21:26:43.605444  2514 tablet.cc:2369] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P c616208720db4b14a9219cae8f2b7d49: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 21:26:43.744082  2242 tablet.cc:2369] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:43.846361  3105 raft_consensus.cc:482] T fe578b007ccd496ba2f70cda8e90c91f P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:43.846742  3105 raft_consensus.cc:504] T fe578b007ccd496ba2f70cda8e90c91f P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:43.847640  3105 leader_election.cc:288] T fe578b007ccd496ba2f70cda8e90c91f P d961985de7874e1990a19bb9b2da14c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687)
I20240502 21:26:43.848728  2467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe578b007ccd496ba2f70cda8e90c91f" candidate_uuid: "d961985de7874e1990a19bb9b2da14c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c616208720db4b14a9219cae8f2b7d49" is_pre_election: true
I20240502 21:26:43.849086  2467 raft_consensus.cc:2454] T fe578b007ccd496ba2f70cda8e90c91f P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d961985de7874e1990a19bb9b2da14c2 in term 0.
I20240502 21:26:43.849833  2128 leader_election.cc:302] T fe578b007ccd496ba2f70cda8e90c91f P d961985de7874e1990a19bb9b2da14c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c616208720db4b14a9219cae8f2b7d49, d961985de7874e1990a19bb9b2da14c2; no voters: 
I20240502 21:26:43.850378  3105 raft_consensus.cc:2781] T fe578b007ccd496ba2f70cda8e90c91f P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:43.850540  3105 raft_consensus.cc:482] T fe578b007ccd496ba2f70cda8e90c91f P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:43.850700  3105 raft_consensus.cc:3037] T fe578b007ccd496ba2f70cda8e90c91f P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:43.853142  3105 raft_consensus.cc:504] T fe578b007ccd496ba2f70cda8e90c91f P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:43.853930  3105 leader_election.cc:288] T fe578b007ccd496ba2f70cda8e90c91f P d961985de7874e1990a19bb9b2da14c2 [CANDIDATE]: Term 1 election: Requested vote from peers c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687)
I20240502 21:26:43.854847  2467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe578b007ccd496ba2f70cda8e90c91f" candidate_uuid: "d961985de7874e1990a19bb9b2da14c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c616208720db4b14a9219cae8f2b7d49"
I20240502 21:26:43.855214  2467 raft_consensus.cc:3037] T fe578b007ccd496ba2f70cda8e90c91f P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:43.857587  2467 raft_consensus.cc:2454] T fe578b007ccd496ba2f70cda8e90c91f P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d961985de7874e1990a19bb9b2da14c2 in term 1.
I20240502 21:26:43.858278  2128 leader_election.cc:302] T fe578b007ccd496ba2f70cda8e90c91f P d961985de7874e1990a19bb9b2da14c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c616208720db4b14a9219cae8f2b7d49, d961985de7874e1990a19bb9b2da14c2; no voters: 
I20240502 21:26:43.858697  3105 raft_consensus.cc:2781] T fe578b007ccd496ba2f70cda8e90c91f P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:43.858903  3105 raft_consensus.cc:686] T fe578b007ccd496ba2f70cda8e90c91f P d961985de7874e1990a19bb9b2da14c2 [term 1 LEADER]: Becoming Leader. State: Replica: d961985de7874e1990a19bb9b2da14c2, State: Running, Role: LEADER
I20240502 21:26:43.859226  3105 consensus_queue.cc:237] T fe578b007ccd496ba2f70cda8e90c91f P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:43.862226  2051 catalog_manager.cc:5484] T fe578b007ccd496ba2f70cda8e90c91f P d961985de7874e1990a19bb9b2da14c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d961985de7874e1990a19bb9b2da14c2 (127.27.196.65). New cstate: current_term: 1 leader_uuid: "d961985de7874e1990a19bb9b2da14c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } }
I20240502 21:26:44.030822  3100 consensus_queue.cc:1035] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:44.090692  3105 consensus_queue.cc:1035] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:44.299183  3118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:44.299621  3118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:44.309584  3118 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:26:44.323036  3113 consensus_queue.cc:1035] T fe578b007ccd496ba2f70cda8e90c91f P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'

INSERT report
    rows total: 2000
    time total: 222.353 ms
  time per row: 0.111177 ms
I20240502 21:26:44.712952  2051 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52214:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 21:26:44.723865  2309 tablet_service.cc:1467] Processing CreateTablet for tablet 99b8f0399d884e41b009da4c3f51fa93 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2c42f3c4552d47169195e69073614d5b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:44.724411  2309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99b8f0399d884e41b009da4c3f51fa93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:44.724439  2447 tablet_service.cc:1467] Processing CreateTablet for tablet 99b8f0399d884e41b009da4c3f51fa93 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2c42f3c4552d47169195e69073614d5b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:44.724896  2447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99b8f0399d884e41b009da4c3f51fa93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:44.730286  3173 tablet_bootstrap.cc:492] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22: Bootstrap starting.
I20240502 21:26:44.730592  2173 tablet_service.cc:1467] Processing CreateTablet for tablet 068c9a3114f648d1a183c9b90f4d365e (DEFAULT_TABLE table=rebalance_test_table_1 [id=2c42f3c4552d47169195e69073614d5b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:44.731135  2173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 068c9a3114f648d1a183c9b90f4d365e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:44.731012  2308 tablet_service.cc:1467] Processing CreateTablet for tablet dc17a741596a4cb59692b7f13cf254e7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2c42f3c4552d47169195e69073614d5b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:44.731828  2308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc17a741596a4cb59692b7f13cf254e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:44.732281  3174 tablet_bootstrap.cc:492] T 99b8f0399d884e41b009da4c3f51fa93 P c616208720db4b14a9219cae8f2b7d49: Bootstrap starting.
I20240502 21:26:44.733601  3173 tablet_bootstrap.cc:654] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:44.734501  3174 tablet_bootstrap.cc:654] T 99b8f0399d884e41b009da4c3f51fa93 P c616208720db4b14a9219cae8f2b7d49: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:44.731128  2307 tablet_service.cc:1467] Processing CreateTablet for tablet 068c9a3114f648d1a183c9b90f4d365e (DEFAULT_TABLE table=rebalance_test_table_1 [id=2c42f3c4552d47169195e69073614d5b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:44.735862  2307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 068c9a3114f648d1a183c9b90f4d365e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:44.736214  2446 tablet_service.cc:1467] Processing CreateTablet for tablet dc17a741596a4cb59692b7f13cf254e7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2c42f3c4552d47169195e69073614d5b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:44.736658  2446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc17a741596a4cb59692b7f13cf254e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:44.749647  3177 tablet_bootstrap.cc:492] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2: Bootstrap starting.
I20240502 21:26:44.750206  3173 tablet_bootstrap.cc:492] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22: No bootstrap required, opened a new log
I20240502 21:26:44.750427  3173 ts_tablet_manager.cc:1367] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22: Time spent bootstrapping tablet: real 0.020s	user 0.013s	sys 0.004s
I20240502 21:26:44.751425  3173 raft_consensus.cc:348] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:44.751674  3173 raft_consensus.cc:374] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.751758  3173 raft_consensus.cc:729] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68f426685b4c4c90bda61ef52177cc22, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.752013  3173 consensus_queue.cc:260] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:44.752254  3177 tablet_bootstrap.cc:654] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:44.753294  3173 ts_tablet_manager.cc:1398] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 21:26:44.753602  3173 tablet_bootstrap.cc:492] T 068c9a3114f648d1a183c9b90f4d365e P 68f426685b4c4c90bda61ef52177cc22: Bootstrap starting.
I20240502 21:26:44.755213  3173 tablet_bootstrap.cc:654] T 068c9a3114f648d1a183c9b90f4d365e P 68f426685b4c4c90bda61ef52177cc22: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:44.755203  3174 tablet_bootstrap.cc:492] T 99b8f0399d884e41b009da4c3f51fa93 P c616208720db4b14a9219cae8f2b7d49: No bootstrap required, opened a new log
I20240502 21:26:44.755385  3174 ts_tablet_manager.cc:1367] T 99b8f0399d884e41b009da4c3f51fa93 P c616208720db4b14a9219cae8f2b7d49: Time spent bootstrapping tablet: real 0.023s	user 0.012s	sys 0.006s
I20240502 21:26:44.756346  3174 raft_consensus.cc:348] T 99b8f0399d884e41b009da4c3f51fa93 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:44.756588  3174 raft_consensus.cc:374] T 99b8f0399d884e41b009da4c3f51fa93 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.756683  3174 raft_consensus.cc:729] T 99b8f0399d884e41b009da4c3f51fa93 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c616208720db4b14a9219cae8f2b7d49, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.756928  3174 consensus_queue.cc:260] T 99b8f0399d884e41b009da4c3f51fa93 P c616208720db4b14a9219cae8f2b7d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:44.757531  3174 ts_tablet_manager.cc:1398] T 99b8f0399d884e41b009da4c3f51fa93 P c616208720db4b14a9219cae8f2b7d49: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 21:26:44.757815  3174 tablet_bootstrap.cc:492] T dc17a741596a4cb59692b7f13cf254e7 P c616208720db4b14a9219cae8f2b7d49: Bootstrap starting.
I20240502 21:26:44.759959  3174 tablet_bootstrap.cc:654] T dc17a741596a4cb59692b7f13cf254e7 P c616208720db4b14a9219cae8f2b7d49: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:44.761524  3177 tablet_bootstrap.cc:492] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2: No bootstrap required, opened a new log
I20240502 21:26:44.762450  3177 ts_tablet_manager.cc:1367] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2: Time spent bootstrapping tablet: real 0.013s	user 0.005s	sys 0.000s
I20240502 21:26:44.763940  3177 raft_consensus.cc:348] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:44.764384  3177 raft_consensus.cc:374] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.764569  3177 raft_consensus.cc:729] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d961985de7874e1990a19bb9b2da14c2, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.766242  3177 consensus_queue.cc:260] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:44.768548  3100 raft_consensus.cc:482] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:44.768781  3100 raft_consensus.cc:504] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:44.770622  3100 leader_election.cc:288] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687)
I20240502 21:26:44.774296  3177 ts_tablet_manager.cc:1398] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2: Time spent starting tablet: real 0.011s	user 0.003s	sys 0.000s
I20240502 21:26:44.785775  3174 tablet_bootstrap.cc:492] T dc17a741596a4cb59692b7f13cf254e7 P c616208720db4b14a9219cae8f2b7d49: No bootstrap required, opened a new log
I20240502 21:26:44.786051  3174 ts_tablet_manager.cc:1367] T dc17a741596a4cb59692b7f13cf254e7 P c616208720db4b14a9219cae8f2b7d49: Time spent bootstrapping tablet: real 0.028s	user 0.010s	sys 0.015s
I20240502 21:26:44.786312  2467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99b8f0399d884e41b009da4c3f51fa93" candidate_uuid: "68f426685b4c4c90bda61ef52177cc22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c616208720db4b14a9219cae8f2b7d49" is_pre_election: true
I20240502 21:26:44.786569  2467 raft_consensus.cc:2454] T 99b8f0399d884e41b009da4c3f51fa93 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68f426685b4c4c90bda61ef52177cc22 in term 0.
I20240502 21:26:44.786831  3173 tablet_bootstrap.cc:492] T 068c9a3114f648d1a183c9b90f4d365e P 68f426685b4c4c90bda61ef52177cc22: No bootstrap required, opened a new log
I20240502 21:26:44.787014  3173 ts_tablet_manager.cc:1367] T 068c9a3114f648d1a183c9b90f4d365e P 68f426685b4c4c90bda61ef52177cc22: Time spent bootstrapping tablet: real 0.033s	user 0.010s	sys 0.019s
I20240502 21:26:44.787045  3174 raft_consensus.cc:348] T dc17a741596a4cb59692b7f13cf254e7 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:44.787279  3174 raft_consensus.cc:374] T dc17a741596a4cb59692b7f13cf254e7 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.787354  3174 raft_consensus.cc:729] T dc17a741596a4cb59692b7f13cf254e7 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c616208720db4b14a9219cae8f2b7d49, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.787331  2263 leader_election.cc:302] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 68f426685b4c4c90bda61ef52177cc22, c616208720db4b14a9219cae8f2b7d49; no voters: 
I20240502 21:26:44.787631  3174 consensus_queue.cc:260] T dc17a741596a4cb59692b7f13cf254e7 P c616208720db4b14a9219cae8f2b7d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:44.787767  3100 raft_consensus.cc:2781] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:44.787883  3100 raft_consensus.cc:482] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:44.787977  3100 raft_consensus.cc:3037] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:44.788262  3174 ts_tablet_manager.cc:1398] T dc17a741596a4cb59692b7f13cf254e7 P c616208720db4b14a9219cae8f2b7d49: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:26:44.790050  3173 raft_consensus.cc:348] T 068c9a3114f648d1a183c9b90f4d365e P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:44.790323  3173 raft_consensus.cc:374] T 068c9a3114f648d1a183c9b90f4d365e P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.790422  3173 raft_consensus.cc:729] T 068c9a3114f648d1a183c9b90f4d365e P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68f426685b4c4c90bda61ef52177cc22, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.790688  3173 consensus_queue.cc:260] T 068c9a3114f648d1a183c9b90f4d365e P 68f426685b4c4c90bda61ef52177cc22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:44.791239  3173 ts_tablet_manager.cc:1398] T 068c9a3114f648d1a183c9b90f4d365e P 68f426685b4c4c90bda61ef52177cc22: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.001s
I20240502 21:26:44.791626  3173 tablet_bootstrap.cc:492] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22: Bootstrap starting.
I20240502 21:26:44.792376  3100 raft_consensus.cc:504] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:44.793434  2467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99b8f0399d884e41b009da4c3f51fa93" candidate_uuid: "68f426685b4c4c90bda61ef52177cc22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c616208720db4b14a9219cae8f2b7d49"
I20240502 21:26:44.793639  2467 raft_consensus.cc:3037] T 99b8f0399d884e41b009da4c3f51fa93 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:44.793954  3173 tablet_bootstrap.cc:654] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:44.796140  3100 leader_election.cc:288] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 1 election: Requested vote from peers c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687)
I20240502 21:26:44.797039  2467 raft_consensus.cc:2454] T 99b8f0399d884e41b009da4c3f51fa93 P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68f426685b4c4c90bda61ef52177cc22 in term 1.
I20240502 21:26:44.797545  2263 leader_election.cc:302] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 68f426685b4c4c90bda61ef52177cc22, c616208720db4b14a9219cae8f2b7d49; no voters: 
I20240502 21:26:44.797772  3173 tablet_bootstrap.cc:492] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22: No bootstrap required, opened a new log
I20240502 21:26:44.797844  3100 raft_consensus.cc:2781] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:44.797948  3173 ts_tablet_manager.cc:1367] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.001s
I20240502 21:26:44.798005  3100 raft_consensus.cc:686] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Becoming Leader. State: Replica: 68f426685b4c4c90bda61ef52177cc22, State: Running, Role: LEADER
I20240502 21:26:44.798274  3100 consensus_queue.cc:237] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:44.799088  3173 raft_consensus.cc:348] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:44.799400  3173 raft_consensus.cc:374] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.799530  3173 raft_consensus.cc:729] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68f426685b4c4c90bda61ef52177cc22, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.799899  3173 consensus_queue.cc:260] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:44.800568  3173 ts_tablet_manager.cc:1398] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.003s
I20240502 21:26:44.800814  2050 catalog_manager.cc:5484] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "68f426685b4c4c90bda61ef52177cc22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } }
I20240502 21:26:44.854431  3100 raft_consensus.cc:482] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:44.854710  3100 raft_consensus.cc:504] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:44.855368  3100 leader_election.cc:288] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13451be0f7134a2c89c0c9330177213e (127.27.196.68:44575)
W20240502 21:26:44.857973  2265 leader_election.cc:334] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 13451be0f7134a2c89c0c9330177213e (127.27.196.68:44575): Network error: Client connection negotiation failed: client connection to 127.27.196.68:44575: connect: Connection refused (error 111)
I20240502 21:26:44.858170  2265 leader_election.cc:302] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 68f426685b4c4c90bda61ef52177cc22; no voters: 13451be0f7134a2c89c0c9330177213e
I20240502 21:26:44.858577  3100 raft_consensus.cc:2726] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:26:44.864097  3113 raft_consensus.cc:482] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:44.864498  3113 raft_consensus.cc:504] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:44.865597  3113 leader_election.cc:288] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253)
I20240502 21:26:44.874143  2330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "068c9a3114f648d1a183c9b90f4d365e" candidate_uuid: "d961985de7874e1990a19bb9b2da14c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68f426685b4c4c90bda61ef52177cc22" is_pre_election: true
I20240502 21:26:44.874567  2330 raft_consensus.cc:2454] T 068c9a3114f648d1a183c9b90f4d365e P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d961985de7874e1990a19bb9b2da14c2 in term 0.
I20240502 21:26:44.875211  2127 leader_election.cc:302] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 68f426685b4c4c90bda61ef52177cc22, d961985de7874e1990a19bb9b2da14c2; no voters: 
I20240502 21:26:44.875618  3113 raft_consensus.cc:2781] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:44.875730  3113 raft_consensus.cc:482] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:44.875881  3113 raft_consensus.cc:3037] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:44.878502  3113 raft_consensus.cc:504] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:44.879346  3113 leader_election.cc:288] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [CANDIDATE]: Term 1 election: Requested vote from peers 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253)
I20240502 21:26:44.880108  2330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "068c9a3114f648d1a183c9b90f4d365e" candidate_uuid: "d961985de7874e1990a19bb9b2da14c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68f426685b4c4c90bda61ef52177cc22"
I20240502 21:26:44.880388  2330 raft_consensus.cc:3037] T 068c9a3114f648d1a183c9b90f4d365e P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:44.882683  2330 raft_consensus.cc:2454] T 068c9a3114f648d1a183c9b90f4d365e P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d961985de7874e1990a19bb9b2da14c2 in term 1.
I20240502 21:26:44.883292  2127 leader_election.cc:302] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 68f426685b4c4c90bda61ef52177cc22, d961985de7874e1990a19bb9b2da14c2; no voters: 
I20240502 21:26:44.883694  3113 raft_consensus.cc:2781] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:44.883916  3113 raft_consensus.cc:686] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [term 1 LEADER]: Becoming Leader. State: Replica: d961985de7874e1990a19bb9b2da14c2, State: Running, Role: LEADER
I20240502 21:26:44.884243  3113 consensus_queue.cc:237] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:44.887251  2050 catalog_manager.cc:5484] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d961985de7874e1990a19bb9b2da14c2 (127.27.196.65). New cstate: current_term: 1 leader_uuid: "d961985de7874e1990a19bb9b2da14c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } }
I20240502 21:26:45.034819  3100 raft_consensus.cc:482] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:45.035064  3100 raft_consensus.cc:504] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:45.035704  3100 leader_election.cc:288] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687)
I20240502 21:26:45.036362  2467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc17a741596a4cb59692b7f13cf254e7" candidate_uuid: "68f426685b4c4c90bda61ef52177cc22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c616208720db4b14a9219cae8f2b7d49" is_pre_election: true
I20240502 21:26:45.036643  2467 raft_consensus.cc:2454] T dc17a741596a4cb59692b7f13cf254e7 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68f426685b4c4c90bda61ef52177cc22 in term 0.
I20240502 21:26:45.037094  2263 leader_election.cc:302] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 68f426685b4c4c90bda61ef52177cc22, c616208720db4b14a9219cae8f2b7d49; no voters: 
I20240502 21:26:45.037449  3100 raft_consensus.cc:2781] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:45.037546  3100 raft_consensus.cc:482] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:45.037662  3100 raft_consensus.cc:3037] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:45.039721  3100 raft_consensus.cc:504] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:45.040212  3100 leader_election.cc:288] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 1 election: Requested vote from peers c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687)
I20240502 21:26:45.040788  2467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc17a741596a4cb59692b7f13cf254e7" candidate_uuid: "68f426685b4c4c90bda61ef52177cc22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c616208720db4b14a9219cae8f2b7d49"
I20240502 21:26:45.041005  2467 raft_consensus.cc:3037] T dc17a741596a4cb59692b7f13cf254e7 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:45.043248  2467 raft_consensus.cc:2454] T dc17a741596a4cb59692b7f13cf254e7 P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68f426685b4c4c90bda61ef52177cc22 in term 1.
I20240502 21:26:45.043812  2263 leader_election.cc:302] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 68f426685b4c4c90bda61ef52177cc22, c616208720db4b14a9219cae8f2b7d49; no voters: 
I20240502 21:26:45.044309  3100 raft_consensus.cc:2781] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:45.044639  3100 raft_consensus.cc:686] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Becoming Leader. State: Replica: 68f426685b4c4c90bda61ef52177cc22, State: Running, Role: LEADER
I20240502 21:26:45.045045  3100 consensus_queue.cc:237] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:45.048808  2050 catalog_manager.cc:5484] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "68f426685b4c4c90bda61ef52177cc22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } }
I20240502 21:26:45.253038  3100 consensus_queue.cc:1035] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:45.412940  3113 consensus_queue.cc:1035] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:45.520743  3191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:45.521139  3191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:45.529263  3191 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 21:26:45.659488  3112 consensus_queue.cc:1035] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 241.782 ms
  time per row: 0.120891 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240502 21:26:46.895754  2050 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52214:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 21:26:46.908689  2173 tablet_service.cc:1467] Processing CreateTablet for tablet d4392d14cf254b72bf506f9c6cfdd55d (DEFAULT_TABLE table=rebalance_test_table_2 [id=dbf5b79f029b456d964098be120a38fd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:46.908975  2446 tablet_service.cc:1467] Processing CreateTablet for tablet d4392d14cf254b72bf506f9c6cfdd55d (DEFAULT_TABLE table=rebalance_test_table_2 [id=dbf5b79f029b456d964098be120a38fd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:46.909241  2173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4392d14cf254b72bf506f9c6cfdd55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.909448  2446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4392d14cf254b72bf506f9c6cfdd55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.909863  2308 tablet_service.cc:1467] Processing CreateTablet for tablet 6a38d2b4246e497fa91d2a0a8b127244 (DEFAULT_TABLE table=rebalance_test_table_2 [id=dbf5b79f029b456d964098be120a38fd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:46.910298  2308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a38d2b4246e497fa91d2a0a8b127244. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.912390  2447 tablet_service.cc:1467] Processing CreateTablet for tablet e93810d704674fc4ad4d96501fb5ea42 (DEFAULT_TABLE table=rebalance_test_table_2 [id=dbf5b79f029b456d964098be120a38fd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:46.912744  2175 tablet_service.cc:1467] Processing CreateTablet for tablet e93810d704674fc4ad4d96501fb5ea42 (DEFAULT_TABLE table=rebalance_test_table_2 [id=dbf5b79f029b456d964098be120a38fd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:46.912861  2447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93810d704674fc4ad4d96501fb5ea42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.913167  2175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93810d704674fc4ad4d96501fb5ea42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.920792  2174 tablet_service.cc:1467] Processing CreateTablet for tablet 6a38d2b4246e497fa91d2a0a8b127244 (DEFAULT_TABLE table=rebalance_test_table_2 [id=dbf5b79f029b456d964098be120a38fd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:46.921290  2174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a38d2b4246e497fa91d2a0a8b127244. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.926218  3248 tablet_bootstrap.cc:492] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22: Bootstrap starting.
I20240502 21:26:46.929477  3248 tablet_bootstrap.cc:654] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.932267  3249 tablet_bootstrap.cc:492] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2: Bootstrap starting.
I20240502 21:26:46.934744  3249 tablet_bootstrap.cc:654] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.934947  3247 tablet_bootstrap.cc:492] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49: Bootstrap starting.
I20240502 21:26:46.937445  3249 tablet_bootstrap.cc:492] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2: No bootstrap required, opened a new log
I20240502 21:26:46.937588  3249 ts_tablet_manager.cc:1367] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.004s
I20240502 21:26:46.937700  3248 tablet_bootstrap.cc:492] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22: No bootstrap required, opened a new log
I20240502 21:26:46.937820  3247 tablet_bootstrap.cc:654] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.937930  3248 ts_tablet_manager.cc:1367] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22: Time spent bootstrapping tablet: real 0.012s	user 0.001s	sys 0.008s
I20240502 21:26:46.938285  3249 raft_consensus.cc:348] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:46.938480  3249 raft_consensus.cc:374] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.938577  3249 raft_consensus.cc:729] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d961985de7874e1990a19bb9b2da14c2, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.938840  3249 consensus_queue.cc:260] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:46.938825  3248 raft_consensus.cc:348] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:46.939064  3248 raft_consensus.cc:374] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.939205  3248 raft_consensus.cc:729] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68f426685b4c4c90bda61ef52177cc22, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.939502  3248 consensus_queue.cc:260] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:46.939673  3249 ts_tablet_manager.cc:1398] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 21:26:46.940007  3249 tablet_bootstrap.cc:492] T d4392d14cf254b72bf506f9c6cfdd55d P d961985de7874e1990a19bb9b2da14c2: Bootstrap starting.
I20240502 21:26:46.940269  3248 ts_tablet_manager.cc:1398] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:26:46.941689  3247 tablet_bootstrap.cc:492] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49: No bootstrap required, opened a new log
I20240502 21:26:46.941872  3247 ts_tablet_manager.cc:1367] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49: Time spent bootstrapping tablet: real 0.007s	user 0.003s	sys 0.000s
I20240502 21:26:46.943315  3249 tablet_bootstrap.cc:654] T d4392d14cf254b72bf506f9c6cfdd55d P d961985de7874e1990a19bb9b2da14c2: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.943734  3247 raft_consensus.cc:348] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:46.943974  3247 raft_consensus.cc:374] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.944108  3247 raft_consensus.cc:729] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c616208720db4b14a9219cae8f2b7d49, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.944408  3247 consensus_queue.cc:260] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:46.945345  3247 ts_tablet_manager.cc:1398] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 21:26:46.945681  3247 tablet_bootstrap.cc:492] T e93810d704674fc4ad4d96501fb5ea42 P c616208720db4b14a9219cae8f2b7d49: Bootstrap starting.
I20240502 21:26:46.947564  3247 tablet_bootstrap.cc:654] T e93810d704674fc4ad4d96501fb5ea42 P c616208720db4b14a9219cae8f2b7d49: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.951644  3249 tablet_bootstrap.cc:492] T d4392d14cf254b72bf506f9c6cfdd55d P d961985de7874e1990a19bb9b2da14c2: No bootstrap required, opened a new log
I20240502 21:26:46.951861  3249 ts_tablet_manager.cc:1367] T d4392d14cf254b72bf506f9c6cfdd55d P d961985de7874e1990a19bb9b2da14c2: Time spent bootstrapping tablet: real 0.012s	user 0.000s	sys 0.009s
I20240502 21:26:46.952586  3249 raft_consensus.cc:348] T d4392d14cf254b72bf506f9c6cfdd55d P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:46.952859  3249 raft_consensus.cc:374] T d4392d14cf254b72bf506f9c6cfdd55d P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.952946  3249 raft_consensus.cc:729] T d4392d14cf254b72bf506f9c6cfdd55d P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d961985de7874e1990a19bb9b2da14c2, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.953186  3249 consensus_queue.cc:260] T d4392d14cf254b72bf506f9c6cfdd55d P d961985de7874e1990a19bb9b2da14c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:46.953747  3249 ts_tablet_manager.cc:1398] T d4392d14cf254b72bf506f9c6cfdd55d P d961985de7874e1990a19bb9b2da14c2: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 21:26:46.953980  3247 tablet_bootstrap.cc:492] T e93810d704674fc4ad4d96501fb5ea42 P c616208720db4b14a9219cae8f2b7d49: No bootstrap required, opened a new log
I20240502 21:26:46.954085  3249 tablet_bootstrap.cc:492] T 6a38d2b4246e497fa91d2a0a8b127244 P d961985de7874e1990a19bb9b2da14c2: Bootstrap starting.
I20240502 21:26:46.954280  3247 ts_tablet_manager.cc:1367] T e93810d704674fc4ad4d96501fb5ea42 P c616208720db4b14a9219cae8f2b7d49: Time spent bootstrapping tablet: real 0.009s	user 0.006s	sys 0.000s
I20240502 21:26:46.955497  3249 tablet_bootstrap.cc:654] T 6a38d2b4246e497fa91d2a0a8b127244 P d961985de7874e1990a19bb9b2da14c2: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.955430  3247 raft_consensus.cc:348] T e93810d704674fc4ad4d96501fb5ea42 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:46.955725  3247 raft_consensus.cc:374] T e93810d704674fc4ad4d96501fb5ea42 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.955821  3247 raft_consensus.cc:729] T e93810d704674fc4ad4d96501fb5ea42 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c616208720db4b14a9219cae8f2b7d49, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.956147  3247 consensus_queue.cc:260] T e93810d704674fc4ad4d96501fb5ea42 P c616208720db4b14a9219cae8f2b7d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:46.957283  3249 tablet_bootstrap.cc:492] T 6a38d2b4246e497fa91d2a0a8b127244 P d961985de7874e1990a19bb9b2da14c2: No bootstrap required, opened a new log
I20240502 21:26:46.957453  3249 ts_tablet_manager.cc:1367] T 6a38d2b4246e497fa91d2a0a8b127244 P d961985de7874e1990a19bb9b2da14c2: Time spent bootstrapping tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 21:26:46.957748  3247 ts_tablet_manager.cc:1398] T e93810d704674fc4ad4d96501fb5ea42 P c616208720db4b14a9219cae8f2b7d49: Time spent starting tablet: real 0.003s	user 0.004s	sys 0.000s
I20240502 21:26:46.958148  3249 raft_consensus.cc:348] T 6a38d2b4246e497fa91d2a0a8b127244 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:46.958381  3249 raft_consensus.cc:374] T 6a38d2b4246e497fa91d2a0a8b127244 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.958514  3249 raft_consensus.cc:729] T 6a38d2b4246e497fa91d2a0a8b127244 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d961985de7874e1990a19bb9b2da14c2, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.959280  3196 raft_consensus.cc:482] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:46.958814  3249 consensus_queue.cc:260] T 6a38d2b4246e497fa91d2a0a8b127244 P d961985de7874e1990a19bb9b2da14c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:46.959491  3196 raft_consensus.cc:504] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:46.960209  3196 leader_election.cc:288] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687)
I20240502 21:26:46.960963  2467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93810d704674fc4ad4d96501fb5ea42" candidate_uuid: "d961985de7874e1990a19bb9b2da14c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c616208720db4b14a9219cae8f2b7d49" is_pre_election: true
I20240502 21:26:46.961246  2467 raft_consensus.cc:2454] T e93810d704674fc4ad4d96501fb5ea42 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d961985de7874e1990a19bb9b2da14c2 in term 0.
I20240502 21:26:46.961865  2128 leader_election.cc:302] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c616208720db4b14a9219cae8f2b7d49, d961985de7874e1990a19bb9b2da14c2; no voters: 
I20240502 21:26:46.962342  3196 raft_consensus.cc:2781] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:46.962484  3196 raft_consensus.cc:482] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:46.962631  3196 raft_consensus.cc:3037] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:46.966074  3196 raft_consensus.cc:504] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:46.966745  3196 leader_election.cc:288] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [CANDIDATE]: Term 1 election: Requested vote from peers c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687)
I20240502 21:26:46.967429  3249 ts_tablet_manager.cc:1398] T 6a38d2b4246e497fa91d2a0a8b127244 P d961985de7874e1990a19bb9b2da14c2: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.003s
I20240502 21:26:46.967834  2467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93810d704674fc4ad4d96501fb5ea42" candidate_uuid: "d961985de7874e1990a19bb9b2da14c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c616208720db4b14a9219cae8f2b7d49"
I20240502 21:26:46.968196  2467 raft_consensus.cc:3037] T e93810d704674fc4ad4d96501fb5ea42 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:46.971798  2467 raft_consensus.cc:2454] T e93810d704674fc4ad4d96501fb5ea42 P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d961985de7874e1990a19bb9b2da14c2 in term 1.
I20240502 21:26:46.972524  2128 leader_election.cc:302] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c616208720db4b14a9219cae8f2b7d49, d961985de7874e1990a19bb9b2da14c2; no voters: 
I20240502 21:26:46.973023  3258 raft_consensus.cc:2781] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:46.973213  3258 raft_consensus.cc:686] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [term 1 LEADER]: Becoming Leader. State: Replica: d961985de7874e1990a19bb9b2da14c2, State: Running, Role: LEADER
I20240502 21:26:46.973632  3258 consensus_queue.cc:237] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:46.980683  2050 catalog_manager.cc:5484] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d961985de7874e1990a19bb9b2da14c2 (127.27.196.65). New cstate: current_term: 1 leader_uuid: "d961985de7874e1990a19bb9b2da14c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } }
I20240502 21:26:47.038852  3233 raft_consensus.cc:482] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:47.039088  3233 raft_consensus.cc:504] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:47.039781  3233 leader_election.cc:288] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025)
I20240502 21:26:47.040644  2195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a38d2b4246e497fa91d2a0a8b127244" candidate_uuid: "68f426685b4c4c90bda61ef52177cc22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d961985de7874e1990a19bb9b2da14c2" is_pre_election: true
I20240502 21:26:47.040944  2195 raft_consensus.cc:2454] T 6a38d2b4246e497fa91d2a0a8b127244 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68f426685b4c4c90bda61ef52177cc22 in term 0.
I20240502 21:26:47.041728  2264 leader_election.cc:302] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 68f426685b4c4c90bda61ef52177cc22, d961985de7874e1990a19bb9b2da14c2; no voters: 
I20240502 21:26:47.042279  3233 raft_consensus.cc:2781] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:47.042423  3233 raft_consensus.cc:482] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:47.042513  3233 raft_consensus.cc:3037] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:47.044873  3233 raft_consensus.cc:504] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:47.045565  3233 leader_election.cc:288] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 1 election: Requested vote from peers d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025)
I20240502 21:26:47.046872  2195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a38d2b4246e497fa91d2a0a8b127244" candidate_uuid: "68f426685b4c4c90bda61ef52177cc22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d961985de7874e1990a19bb9b2da14c2"
I20240502 21:26:47.047263  2195 raft_consensus.cc:3037] T 6a38d2b4246e497fa91d2a0a8b127244 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:47.049845  2195 raft_consensus.cc:2454] T 6a38d2b4246e497fa91d2a0a8b127244 P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68f426685b4c4c90bda61ef52177cc22 in term 1.
I20240502 21:26:47.050619  2264 leader_election.cc:302] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 68f426685b4c4c90bda61ef52177cc22, d961985de7874e1990a19bb9b2da14c2; no voters: 
I20240502 21:26:47.051334  3233 raft_consensus.cc:2781] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:47.052033  3233 raft_consensus.cc:686] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Becoming Leader. State: Replica: 68f426685b4c4c90bda61ef52177cc22, State: Running, Role: LEADER
I20240502 21:26:47.052819  3233 consensus_queue.cc:237] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:47.057327  2050 catalog_manager.cc:5484] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "68f426685b4c4c90bda61ef52177cc22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } }
I20240502 21:26:47.286948  3254 raft_consensus.cc:482] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:47.287304  3254 raft_consensus.cc:504] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:47.289232  3254 leader_election.cc:288] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025)
I20240502 21:26:47.297513  2195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4392d14cf254b72bf506f9c6cfdd55d" candidate_uuid: "c616208720db4b14a9219cae8f2b7d49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d961985de7874e1990a19bb9b2da14c2" is_pre_election: true
I20240502 21:26:47.297816  2195 raft_consensus.cc:2454] T d4392d14cf254b72bf506f9c6cfdd55d P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c616208720db4b14a9219cae8f2b7d49 in term 0.
I20240502 21:26:47.298487  2400 leader_election.cc:302] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c616208720db4b14a9219cae8f2b7d49, d961985de7874e1990a19bb9b2da14c2; no voters: 
I20240502 21:26:47.298987  3254 raft_consensus.cc:2781] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:47.299152  3254 raft_consensus.cc:482] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:47.299371  3254 raft_consensus.cc:3037] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:47.301707  3254 raft_consensus.cc:504] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:47.302384  3254 leader_election.cc:288] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [CANDIDATE]: Term 1 election: Requested vote from peers d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025)
I20240502 21:26:47.303164  2195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4392d14cf254b72bf506f9c6cfdd55d" candidate_uuid: "c616208720db4b14a9219cae8f2b7d49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d961985de7874e1990a19bb9b2da14c2"
I20240502 21:26:47.303400  2195 raft_consensus.cc:3037] T d4392d14cf254b72bf506f9c6cfdd55d P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:47.305716  2195 raft_consensus.cc:2454] T d4392d14cf254b72bf506f9c6cfdd55d P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c616208720db4b14a9219cae8f2b7d49 in term 1.
I20240502 21:26:47.306480  2400 leader_election.cc:302] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c616208720db4b14a9219cae8f2b7d49, d961985de7874e1990a19bb9b2da14c2; no voters: 
I20240502 21:26:47.307049  3254 raft_consensus.cc:2781] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:47.307581  3254 raft_consensus.cc:686] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [term 1 LEADER]: Becoming Leader. State: Replica: c616208720db4b14a9219cae8f2b7d49, State: Running, Role: LEADER
I20240502 21:26:47.308143  3254 consensus_queue.cc:237] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:47.311803  2050 catalog_manager.cc:5484] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 reported cstate change: term changed from 0 to 1, leader changed from <none> to c616208720db4b14a9219cae8f2b7d49 (127.27.196.67). New cstate: current_term: 1 leader_uuid: "c616208720db4b14a9219cae8f2b7d49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } }
I20240502 21:26:47.550194  3258 consensus_queue.cc:1035] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:47.623978  3233 consensus_queue.cc:1035] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:47.753472  3266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:47.753825  3266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:47.761670  3266 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:26:47.768410  3254 consensus_queue.cc:1035] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'

INSERT report
    rows total: 2000
    time total: 145.72 ms
  time per row: 0.07286 ms
I20240502 21:26:48.076514  2047 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52214:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 21:26:48.087977  2308 tablet_service.cc:1467] Processing CreateTablet for tablet 2779949906ff486da691638de7374050 (DEFAULT_TABLE table=rebalance_test_table_3 [id=72576758fa73429dba7089fdaa3f9ea3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:48.087980  2446 tablet_service.cc:1467] Processing CreateTablet for tablet 2779949906ff486da691638de7374050 (DEFAULT_TABLE table=rebalance_test_table_3 [id=72576758fa73429dba7089fdaa3f9ea3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:48.088265  2447 tablet_service.cc:1467] Processing CreateTablet for tablet d64bd5e7c22749078d82ad8da12d75dd (DEFAULT_TABLE table=rebalance_test_table_3 [id=72576758fa73429dba7089fdaa3f9ea3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:48.088506  2308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2779949906ff486da691638de7374050. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:48.088549  2446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2779949906ff486da691638de7374050. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:48.089375  2309 tablet_service.cc:1467] Processing CreateTablet for tablet 3a6da907ad0447b3ad2f66f50cd7289d (DEFAULT_TABLE table=rebalance_test_table_3 [id=72576758fa73429dba7089fdaa3f9ea3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:48.089437  2174 tablet_service.cc:1467] Processing CreateTablet for tablet 3a6da907ad0447b3ad2f66f50cd7289d (DEFAULT_TABLE table=rebalance_test_table_3 [id=72576758fa73429dba7089fdaa3f9ea3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:48.089793  2309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a6da907ad0447b3ad2f66f50cd7289d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:48.089859  2174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a6da907ad0447b3ad2f66f50cd7289d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:48.091562  2447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d64bd5e7c22749078d82ad8da12d75dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:48.095852  2307 tablet_service.cc:1467] Processing CreateTablet for tablet d64bd5e7c22749078d82ad8da12d75dd (DEFAULT_TABLE table=rebalance_test_table_3 [id=72576758fa73429dba7089fdaa3f9ea3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:48.096534  2307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d64bd5e7c22749078d82ad8da12d75dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:48.103507  3325 tablet_bootstrap.cc:492] T 2779949906ff486da691638de7374050 P c616208720db4b14a9219cae8f2b7d49: Bootstrap starting.
I20240502 21:26:48.105391  3325 tablet_bootstrap.cc:654] T 2779949906ff486da691638de7374050 P c616208720db4b14a9219cae8f2b7d49: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:48.107695  3326 tablet_bootstrap.cc:492] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22: Bootstrap starting.
I20240502 21:26:48.108464  3325 tablet_bootstrap.cc:492] T 2779949906ff486da691638de7374050 P c616208720db4b14a9219cae8f2b7d49: No bootstrap required, opened a new log
I20240502 21:26:48.108718  3325 ts_tablet_manager.cc:1367] T 2779949906ff486da691638de7374050 P c616208720db4b14a9219cae8f2b7d49: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.005s
I20240502 21:26:48.108909  3327 tablet_bootstrap.cc:492] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2: Bootstrap starting.
I20240502 21:26:48.109963  3326 tablet_bootstrap.cc:654] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:48.110522  3325 raft_consensus.cc:348] T 2779949906ff486da691638de7374050 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:48.110875  3325 raft_consensus.cc:374] T 2779949906ff486da691638de7374050 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:48.111020  3327 tablet_bootstrap.cc:654] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:48.111050  3325 raft_consensus.cc:729] T 2779949906ff486da691638de7374050 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c616208720db4b14a9219cae8f2b7d49, State: Initialized, Role: FOLLOWER
I20240502 21:26:48.111511  3325 consensus_queue.cc:260] T 2779949906ff486da691638de7374050 P c616208720db4b14a9219cae8f2b7d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:48.112325  3325 ts_tablet_manager.cc:1398] T 2779949906ff486da691638de7374050 P c616208720db4b14a9219cae8f2b7d49: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 21:26:48.112380  3326 tablet_bootstrap.cc:492] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22: No bootstrap required, opened a new log
I20240502 21:26:48.112531  3326 ts_tablet_manager.cc:1367] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 21:26:48.112923  3325 tablet_bootstrap.cc:492] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49: Bootstrap starting.
I20240502 21:26:48.113581  3326 raft_consensus.cc:348] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:48.113827  3326 raft_consensus.cc:374] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:48.113938  3326 raft_consensus.cc:729] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68f426685b4c4c90bda61ef52177cc22, State: Initialized, Role: FOLLOWER
I20240502 21:26:48.114318  3326 consensus_queue.cc:260] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:48.115160  3325 tablet_bootstrap.cc:654] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:48.115139  3326 ts_tablet_manager.cc:1398] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 21:26:48.115916  3326 tablet_bootstrap.cc:492] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22: Bootstrap starting.
I20240502 21:26:48.118135  3326 tablet_bootstrap.cc:654] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:48.120841  3327 tablet_bootstrap.cc:492] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2: No bootstrap required, opened a new log
I20240502 21:26:48.121058  3327 ts_tablet_manager.cc:1367] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2: Time spent bootstrapping tablet: real 0.012s	user 0.004s	sys 0.005s
I20240502 21:26:48.122834  3327 raft_consensus.cc:348] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:48.123090  3327 raft_consensus.cc:374] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:48.123225  3327 raft_consensus.cc:729] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d961985de7874e1990a19bb9b2da14c2, State: Initialized, Role: FOLLOWER
I20240502 21:26:48.123504  3327 consensus_queue.cc:260] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:48.125095  3327 ts_tablet_manager.cc:1398] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.003s
I20240502 21:26:48.127085  3325 tablet_bootstrap.cc:492] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49: No bootstrap required, opened a new log
I20240502 21:26:48.127195  3326 tablet_bootstrap.cc:492] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22: No bootstrap required, opened a new log
I20240502 21:26:48.127310  3325 ts_tablet_manager.cc:1367] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49: Time spent bootstrapping tablet: real 0.014s	user 0.007s	sys 0.006s
I20240502 21:26:48.127377  3326 ts_tablet_manager.cc:1367] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.007s
I20240502 21:26:48.128151  3325 raft_consensus.cc:348] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:48.128399  3325 raft_consensus.cc:374] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:48.128489  3325 raft_consensus.cc:729] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c616208720db4b14a9219cae8f2b7d49, State: Initialized, Role: FOLLOWER
I20240502 21:26:48.128417  3326 raft_consensus.cc:348] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:48.128695  3326 raft_consensus.cc:374] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:48.128798  3326 raft_consensus.cc:729] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68f426685b4c4c90bda61ef52177cc22, State: Initialized, Role: FOLLOWER
I20240502 21:26:48.128774  3325 consensus_queue.cc:260] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:48.129282  3326 consensus_queue.cc:260] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:48.130254  3326 ts_tablet_manager.cc:1398] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.000s
I20240502 21:26:48.130599  3325 ts_tablet_manager.cc:1398] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 21:26:48.130667  3326 tablet_bootstrap.cc:492] T d64bd5e7c22749078d82ad8da12d75dd P 68f426685b4c4c90bda61ef52177cc22: Bootstrap starting.
I20240502 21:26:48.133298  3326 tablet_bootstrap.cc:654] T d64bd5e7c22749078d82ad8da12d75dd P 68f426685b4c4c90bda61ef52177cc22: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:48.136130  3326 tablet_bootstrap.cc:492] T d64bd5e7c22749078d82ad8da12d75dd P 68f426685b4c4c90bda61ef52177cc22: No bootstrap required, opened a new log
I20240502 21:26:48.136319  3326 ts_tablet_manager.cc:1367] T d64bd5e7c22749078d82ad8da12d75dd P 68f426685b4c4c90bda61ef52177cc22: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240502 21:26:48.137094  3326 raft_consensus.cc:348] T d64bd5e7c22749078d82ad8da12d75dd P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:48.137310  3326 raft_consensus.cc:374] T d64bd5e7c22749078d82ad8da12d75dd P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:48.137420  3326 raft_consensus.cc:729] T d64bd5e7c22749078d82ad8da12d75dd P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68f426685b4c4c90bda61ef52177cc22, State: Initialized, Role: FOLLOWER
I20240502 21:26:48.137697  3326 consensus_queue.cc:260] T d64bd5e7c22749078d82ad8da12d75dd P 68f426685b4c4c90bda61ef52177cc22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:48.138185  3326 ts_tablet_manager.cc:1398] T d64bd5e7c22749078d82ad8da12d75dd P 68f426685b4c4c90bda61ef52177cc22: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 21:26:48.312891  3233 raft_consensus.cc:482] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:48.313115  3233 raft_consensus.cc:504] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:48.313767  3233 leader_election.cc:288] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13451be0f7134a2c89c0c9330177213e (127.27.196.68:44575)
W20240502 21:26:48.315460  2265 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.68:44575: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240502 21:26:48.317701  2265 leader_election.cc:334] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 13451be0f7134a2c89c0c9330177213e (127.27.196.68:44575): Network error: Client connection negotiation failed: client connection to 127.27.196.68:44575: connect: Connection refused (error 111)
I20240502 21:26:48.317838  2265 leader_election.cc:302] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 68f426685b4c4c90bda61ef52177cc22; no voters: 13451be0f7134a2c89c0c9330177213e
I20240502 21:26:48.318218  3233 raft_consensus.cc:2726] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:26:48.387382  3233 raft_consensus.cc:482] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:48.387723  3233 raft_consensus.cc:504] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:48.388532  3233 leader_election.cc:288] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687)
I20240502 21:26:48.389470  2467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2779949906ff486da691638de7374050" candidate_uuid: "68f426685b4c4c90bda61ef52177cc22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c616208720db4b14a9219cae8f2b7d49" is_pre_election: true
I20240502 21:26:48.389806  2467 raft_consensus.cc:2454] T 2779949906ff486da691638de7374050 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68f426685b4c4c90bda61ef52177cc22 in term 0.
I20240502 21:26:48.390538  2263 leader_election.cc:302] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 68f426685b4c4c90bda61ef52177cc22, c616208720db4b14a9219cae8f2b7d49; no voters: 
I20240502 21:26:48.390985  3233 raft_consensus.cc:2781] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:48.391176  3233 raft_consensus.cc:482] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:48.391266  3233 raft_consensus.cc:3037] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:48.393522  3233 raft_consensus.cc:504] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:48.394155  3233 leader_election.cc:288] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 1 election: Requested vote from peers c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687)
I20240502 21:26:48.394933  2467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2779949906ff486da691638de7374050" candidate_uuid: "68f426685b4c4c90bda61ef52177cc22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c616208720db4b14a9219cae8f2b7d49"
I20240502 21:26:48.395283  2467 raft_consensus.cc:3037] T 2779949906ff486da691638de7374050 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:48.397289  2467 raft_consensus.cc:2454] T 2779949906ff486da691638de7374050 P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68f426685b4c4c90bda61ef52177cc22 in term 1.
I20240502 21:26:48.397816  2263 leader_election.cc:302] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 68f426685b4c4c90bda61ef52177cc22, c616208720db4b14a9219cae8f2b7d49; no voters: 
I20240502 21:26:48.398368  3233 raft_consensus.cc:2781] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:48.398623  3233 raft_consensus.cc:686] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Becoming Leader. State: Replica: 68f426685b4c4c90bda61ef52177cc22, State: Running, Role: LEADER
I20240502 21:26:48.399035  3233 consensus_queue.cc:237] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:48.402768  2048 catalog_manager.cc:5484] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "68f426685b4c4c90bda61ef52177cc22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } }
I20240502 21:26:48.444963  3233 raft_consensus.cc:482] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:48.445209  3233 raft_consensus.cc:504] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:48.445873  3233 leader_election.cc:288] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025)
I20240502 21:26:48.446574  2195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a6da907ad0447b3ad2f66f50cd7289d" candidate_uuid: "68f426685b4c4c90bda61ef52177cc22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d961985de7874e1990a19bb9b2da14c2" is_pre_election: true
I20240502 21:26:48.446843  2195 raft_consensus.cc:2454] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68f426685b4c4c90bda61ef52177cc22 in term 0.
I20240502 21:26:48.447600  2264 leader_election.cc:302] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 68f426685b4c4c90bda61ef52177cc22, d961985de7874e1990a19bb9b2da14c2; no voters: 
I20240502 21:26:48.448149  3233 raft_consensus.cc:2781] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:48.448257  3233 raft_consensus.cc:482] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:48.448371  3233 raft_consensus.cc:3037] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:48.452399  3233 raft_consensus.cc:504] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:48.453269  3233 leader_election.cc:288] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 1 election: Requested vote from peers d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025)
I20240502 21:26:48.454159  2195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a6da907ad0447b3ad2f66f50cd7289d" candidate_uuid: "68f426685b4c4c90bda61ef52177cc22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d961985de7874e1990a19bb9b2da14c2"
I20240502 21:26:48.454408  2195 raft_consensus.cc:3037] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:48.456521  2195 raft_consensus.cc:2454] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68f426685b4c4c90bda61ef52177cc22 in term 1.
I20240502 21:26:48.457048  2264 leader_election.cc:302] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 68f426685b4c4c90bda61ef52177cc22, d961985de7874e1990a19bb9b2da14c2; no voters: 
I20240502 21:26:48.457526  3233 raft_consensus.cc:2781] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:48.457790  3233 raft_consensus.cc:686] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Becoming Leader. State: Replica: 68f426685b4c4c90bda61ef52177cc22, State: Running, Role: LEADER
I20240502 21:26:48.458118  3233 consensus_queue.cc:237] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:48.461584  2048 catalog_manager.cc:5484] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "68f426685b4c4c90bda61ef52177cc22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } }
I20240502 21:26:48.617321  3254 raft_consensus.cc:482] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:48.617553  3254 raft_consensus.cc:504] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:48.618288  3254 leader_election.cc:288] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253)
I20240502 21:26:48.625864  2330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d64bd5e7c22749078d82ad8da12d75dd" candidate_uuid: "c616208720db4b14a9219cae8f2b7d49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68f426685b4c4c90bda61ef52177cc22" is_pre_election: true
I20240502 21:26:48.626157  2330 raft_consensus.cc:2454] T d64bd5e7c22749078d82ad8da12d75dd P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c616208720db4b14a9219cae8f2b7d49 in term 0.
I20240502 21:26:48.626763  2398 leader_election.cc:302] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 68f426685b4c4c90bda61ef52177cc22, c616208720db4b14a9219cae8f2b7d49; no voters: 
I20240502 21:26:48.627157  3254 raft_consensus.cc:2781] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:48.627333  3254 raft_consensus.cc:482] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:48.627437  3254 raft_consensus.cc:3037] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:48.629514  3254 raft_consensus.cc:504] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:48.630076  3254 leader_election.cc:288] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49 [CANDIDATE]: Term 1 election: Requested vote from peers 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253)
I20240502 21:26:48.630824  2330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d64bd5e7c22749078d82ad8da12d75dd" candidate_uuid: "c616208720db4b14a9219cae8f2b7d49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68f426685b4c4c90bda61ef52177cc22"
I20240502 21:26:48.631048  2330 raft_consensus.cc:3037] T d64bd5e7c22749078d82ad8da12d75dd P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:48.633252  2330 raft_consensus.cc:2454] T d64bd5e7c22749078d82ad8da12d75dd P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c616208720db4b14a9219cae8f2b7d49 in term 1.
I20240502 21:26:48.634033  2398 leader_election.cc:302] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 68f426685b4c4c90bda61ef52177cc22, c616208720db4b14a9219cae8f2b7d49; no voters: 
I20240502 21:26:48.634536  3254 raft_consensus.cc:2781] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:48.635007  3254 raft_consensus.cc:686] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49 [term 1 LEADER]: Becoming Leader. State: Replica: c616208720db4b14a9219cae8f2b7d49, State: Running, Role: LEADER
I20240502 21:26:48.635382  3254 consensus_queue.cc:237] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } }
I20240502 21:26:48.638274  2048 catalog_manager.cc:5484] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49 reported cstate change: term changed from 0 to 1, leader changed from <none> to c616208720db4b14a9219cae8f2b7d49 (127.27.196.67). New cstate: current_term: 1 leader_uuid: "c616208720db4b14a9219cae8f2b7d49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } }
I20240502 21:26:48.847973  3233 consensus_queue.cc:1035] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:48.894104  3272 consensus_queue.cc:1035] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:49.070850  3342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:49.071285  3342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:49.079257  3342 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:26:49.112620  3254 consensus_queue.cc:1035] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'

INSERT report
    rows total: 2000
    time total: 175.181 ms
  time per row: 0.0875906 ms
I20240502 21:26:49.427877  2043 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52214:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 21:26:49.438789  2446 tablet_service.cc:1467] Processing CreateTablet for tablet 0306f5bfea0a49eca06554a40ba50d34 (DEFAULT_TABLE table=rebalance_test_table_4 [id=69e9b99652e74772bfdd02eeaf9e1bd1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:49.439381  2446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0306f5bfea0a49eca06554a40ba50d34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:49.439741  2308 tablet_service.cc:1467] Processing CreateTablet for tablet be2cbcd79c1e40f49701271bb247f4d0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=69e9b99652e74772bfdd02eeaf9e1bd1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:49.439991  2447 tablet_service.cc:1467] Processing CreateTablet for tablet 0264c436aafa422a9a54599e74e0221c (DEFAULT_TABLE table=rebalance_test_table_4 [id=69e9b99652e74772bfdd02eeaf9e1bd1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:49.440217  2308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be2cbcd79c1e40f49701271bb247f4d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:49.440471  2447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0264c436aafa422a9a54599e74e0221c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:49.440512  2175 tablet_service.cc:1467] Processing CreateTablet for tablet be2cbcd79c1e40f49701271bb247f4d0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=69e9b99652e74772bfdd02eeaf9e1bd1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:49.440945  2175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be2cbcd79c1e40f49701271bb247f4d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:49.443612  2174 tablet_service.cc:1467] Processing CreateTablet for tablet 0306f5bfea0a49eca06554a40ba50d34 (DEFAULT_TABLE table=rebalance_test_table_4 [id=69e9b99652e74772bfdd02eeaf9e1bd1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:49.444108  2174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0306f5bfea0a49eca06554a40ba50d34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:49.447678  2173 tablet_service.cc:1467] Processing CreateTablet for tablet 0264c436aafa422a9a54599e74e0221c (DEFAULT_TABLE table=rebalance_test_table_4 [id=69e9b99652e74772bfdd02eeaf9e1bd1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:49.448184  2173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0264c436aafa422a9a54599e74e0221c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:49.452412  3401 tablet_bootstrap.cc:492] T be2cbcd79c1e40f49701271bb247f4d0 P d961985de7874e1990a19bb9b2da14c2: Bootstrap starting.
I20240502 21:26:49.455076  3401 tablet_bootstrap.cc:654] T be2cbcd79c1e40f49701271bb247f4d0 P d961985de7874e1990a19bb9b2da14c2: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:49.464963  3402 tablet_bootstrap.cc:492] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22: Bootstrap starting.
I20240502 21:26:49.465317  3401 tablet_bootstrap.cc:492] T be2cbcd79c1e40f49701271bb247f4d0 P d961985de7874e1990a19bb9b2da14c2: No bootstrap required, opened a new log
I20240502 21:26:49.465538  3401 ts_tablet_manager.cc:1367] T be2cbcd79c1e40f49701271bb247f4d0 P d961985de7874e1990a19bb9b2da14c2: Time spent bootstrapping tablet: real 0.013s	user 0.001s	sys 0.009s
I20240502 21:26:49.466485  3401 raft_consensus.cc:348] T be2cbcd79c1e40f49701271bb247f4d0 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:49.466746  3401 raft_consensus.cc:374] T be2cbcd79c1e40f49701271bb247f4d0 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:49.466850  3401 raft_consensus.cc:729] T be2cbcd79c1e40f49701271bb247f4d0 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d961985de7874e1990a19bb9b2da14c2, State: Initialized, Role: FOLLOWER
I20240502 21:26:49.467063  3402 tablet_bootstrap.cc:654] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:49.467178  3401 consensus_queue.cc:260] T be2cbcd79c1e40f49701271bb247f4d0 P d961985de7874e1990a19bb9b2da14c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:49.467687  3400 tablet_bootstrap.cc:492] T 0264c436aafa422a9a54599e74e0221c P c616208720db4b14a9219cae8f2b7d49: Bootstrap starting.
I20240502 21:26:49.467893  3401 ts_tablet_manager.cc:1398] T be2cbcd79c1e40f49701271bb247f4d0 P d961985de7874e1990a19bb9b2da14c2: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 21:26:49.468282  3401 tablet_bootstrap.cc:492] T 0306f5bfea0a49eca06554a40ba50d34 P d961985de7874e1990a19bb9b2da14c2: Bootstrap starting.
I20240502 21:26:49.470351  3401 tablet_bootstrap.cc:654] T 0306f5bfea0a49eca06554a40ba50d34 P d961985de7874e1990a19bb9b2da14c2: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:49.470336  3400 tablet_bootstrap.cc:654] T 0264c436aafa422a9a54599e74e0221c P c616208720db4b14a9219cae8f2b7d49: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:49.476533  3401 tablet_bootstrap.cc:492] T 0306f5bfea0a49eca06554a40ba50d34 P d961985de7874e1990a19bb9b2da14c2: No bootstrap required, opened a new log
I20240502 21:26:49.476881  3401 ts_tablet_manager.cc:1367] T 0306f5bfea0a49eca06554a40ba50d34 P d961985de7874e1990a19bb9b2da14c2: Time spent bootstrapping tablet: real 0.009s	user 0.006s	sys 0.001s
I20240502 21:26:49.478261  3401 raft_consensus.cc:348] T 0306f5bfea0a49eca06554a40ba50d34 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:49.478608  3401 raft_consensus.cc:374] T 0306f5bfea0a49eca06554a40ba50d34 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:49.478724  3401 raft_consensus.cc:729] T 0306f5bfea0a49eca06554a40ba50d34 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d961985de7874e1990a19bb9b2da14c2, State: Initialized, Role: FOLLOWER
I20240502 21:26:49.479033  3401 consensus_queue.cc:260] T 0306f5bfea0a49eca06554a40ba50d34 P d961985de7874e1990a19bb9b2da14c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:49.479777  3401 ts_tablet_manager.cc:1398] T 0306f5bfea0a49eca06554a40ba50d34 P d961985de7874e1990a19bb9b2da14c2: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 21:26:49.480150  3401 tablet_bootstrap.cc:492] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2: Bootstrap starting.
I20240502 21:26:49.481479  3402 tablet_bootstrap.cc:492] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22: No bootstrap required, opened a new log
I20240502 21:26:49.481479  3400 tablet_bootstrap.cc:492] T 0264c436aafa422a9a54599e74e0221c P c616208720db4b14a9219cae8f2b7d49: No bootstrap required, opened a new log
I20240502 21:26:49.481683  3400 ts_tablet_manager.cc:1367] T 0264c436aafa422a9a54599e74e0221c P c616208720db4b14a9219cae8f2b7d49: Time spent bootstrapping tablet: real 0.014s	user 0.004s	sys 0.007s
I20240502 21:26:49.481804  3402 ts_tablet_manager.cc:1367] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22: Time spent bootstrapping tablet: real 0.017s	user 0.008s	sys 0.004s
I20240502 21:26:49.482259  3401 tablet_bootstrap.cc:654] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:49.482990  3400 raft_consensus.cc:348] T 0264c436aafa422a9a54599e74e0221c P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:49.483031  3402 raft_consensus.cc:348] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:49.483352  3402 raft_consensus.cc:374] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:49.483466  3400 raft_consensus.cc:374] T 0264c436aafa422a9a54599e74e0221c P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:49.483502  3402 raft_consensus.cc:729] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68f426685b4c4c90bda61ef52177cc22, State: Initialized, Role: FOLLOWER
I20240502 21:26:49.483639  3400 raft_consensus.cc:729] T 0264c436aafa422a9a54599e74e0221c P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c616208720db4b14a9219cae8f2b7d49, State: Initialized, Role: FOLLOWER
I20240502 21:26:49.483937  3400 consensus_queue.cc:260] T 0264c436aafa422a9a54599e74e0221c P c616208720db4b14a9219cae8f2b7d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:49.484006  3402 consensus_queue.cc:260] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:49.484884  3400 ts_tablet_manager.cc:1398] T 0264c436aafa422a9a54599e74e0221c P c616208720db4b14a9219cae8f2b7d49: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 21:26:49.485333  3400 tablet_bootstrap.cc:492] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49: Bootstrap starting.
I20240502 21:26:49.485909  3402 ts_tablet_manager.cc:1398] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 21:26:49.487955  3400 tablet_bootstrap.cc:654] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:49.492446  3401 tablet_bootstrap.cc:492] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2: No bootstrap required, opened a new log
I20240502 21:26:49.493664  3401 ts_tablet_manager.cc:1367] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2: Time spent bootstrapping tablet: real 0.014s	user 0.005s	sys 0.000s
I20240502 21:26:49.494753  3401 raft_consensus.cc:348] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:49.495021  3401 raft_consensus.cc:374] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:49.495177  3401 raft_consensus.cc:729] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d961985de7874e1990a19bb9b2da14c2, State: Initialized, Role: FOLLOWER
I20240502 21:26:49.495525  3401 consensus_queue.cc:260] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:49.496723  3401 ts_tablet_manager.cc:1398] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 21:26:49.497308  3400 tablet_bootstrap.cc:492] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49: No bootstrap required, opened a new log
I20240502 21:26:49.497531  3400 ts_tablet_manager.cc:1367] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49: Time spent bootstrapping tablet: real 0.012s	user 0.005s	sys 0.000s
I20240502 21:26:49.498283  3400 raft_consensus.cc:348] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:49.498508  3400 raft_consensus.cc:374] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:49.498629  3400 raft_consensus.cc:729] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c616208720db4b14a9219cae8f2b7d49, State: Initialized, Role: FOLLOWER
I20240502 21:26:49.499024  3400 consensus_queue.cc:260] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:49.500494  3400 ts_tablet_manager.cc:1398] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 21:26:49.593593  3268 raft_consensus.cc:482] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:49.593945  3268 raft_consensus.cc:504] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:49.594779  3268 leader_election.cc:288] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687)
I20240502 21:26:49.595729  2467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0264c436aafa422a9a54599e74e0221c" candidate_uuid: "d961985de7874e1990a19bb9b2da14c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c616208720db4b14a9219cae8f2b7d49" is_pre_election: true
I20240502 21:26:49.596020  2467 raft_consensus.cc:2454] T 0264c436aafa422a9a54599e74e0221c P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d961985de7874e1990a19bb9b2da14c2 in term 0.
I20240502 21:26:49.596742  2128 leader_election.cc:302] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c616208720db4b14a9219cae8f2b7d49, d961985de7874e1990a19bb9b2da14c2; no voters: 
I20240502 21:26:49.597157  3268 raft_consensus.cc:2781] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:49.597303  3268 raft_consensus.cc:482] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:49.597417  3268 raft_consensus.cc:3037] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:49.599962  3268 raft_consensus.cc:504] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:49.600607  3268 leader_election.cc:288] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2 [CANDIDATE]: Term 1 election: Requested vote from peers c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687)
I20240502 21:26:49.601528  2467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0264c436aafa422a9a54599e74e0221c" candidate_uuid: "d961985de7874e1990a19bb9b2da14c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c616208720db4b14a9219cae8f2b7d49"
I20240502 21:26:49.601833  2467 raft_consensus.cc:3037] T 0264c436aafa422a9a54599e74e0221c P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:49.605561  2467 raft_consensus.cc:2454] T 0264c436aafa422a9a54599e74e0221c P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d961985de7874e1990a19bb9b2da14c2 in term 1.
I20240502 21:26:49.606217  2128 leader_election.cc:302] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c616208720db4b14a9219cae8f2b7d49, d961985de7874e1990a19bb9b2da14c2; no voters: 
I20240502 21:26:49.606693  3268 raft_consensus.cc:2781] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:49.607223  3268 raft_consensus.cc:686] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2 [term 1 LEADER]: Becoming Leader. State: Replica: d961985de7874e1990a19bb9b2da14c2, State: Running, Role: LEADER
I20240502 21:26:49.607617  3268 consensus_queue.cc:237] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:49.611704  2047 catalog_manager.cc:5484] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d961985de7874e1990a19bb9b2da14c2 (127.27.196.65). New cstate: current_term: 1 leader_uuid: "d961985de7874e1990a19bb9b2da14c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } }
I20240502 21:26:49.620317  3254 raft_consensus.cc:482] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:49.620527  3254 raft_consensus.cc:504] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:49.621110  3254 leader_election.cc:288] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025)
I20240502 21:26:49.621775  2194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0306f5bfea0a49eca06554a40ba50d34" candidate_uuid: "c616208720db4b14a9219cae8f2b7d49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d961985de7874e1990a19bb9b2da14c2" is_pre_election: true
I20240502 21:26:49.622011  2194 raft_consensus.cc:2454] T 0306f5bfea0a49eca06554a40ba50d34 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c616208720db4b14a9219cae8f2b7d49 in term 0.
I20240502 21:26:49.622522  2400 leader_election.cc:302] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c616208720db4b14a9219cae8f2b7d49, d961985de7874e1990a19bb9b2da14c2; no voters: 
I20240502 21:26:49.622825  3254 raft_consensus.cc:2781] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:49.622967  3254 raft_consensus.cc:482] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:49.623067  3254 raft_consensus.cc:3037] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:49.625310  3254 raft_consensus.cc:504] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:49.625939  3254 leader_election.cc:288] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [CANDIDATE]: Term 1 election: Requested vote from peers d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025)
I20240502 21:26:49.626821  2194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0306f5bfea0a49eca06554a40ba50d34" candidate_uuid: "c616208720db4b14a9219cae8f2b7d49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d961985de7874e1990a19bb9b2da14c2"
I20240502 21:26:49.627060  2194 raft_consensus.cc:3037] T 0306f5bfea0a49eca06554a40ba50d34 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:49.629098  2194 raft_consensus.cc:2454] T 0306f5bfea0a49eca06554a40ba50d34 P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c616208720db4b14a9219cae8f2b7d49 in term 1.
I20240502 21:26:49.629611  2400 leader_election.cc:302] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c616208720db4b14a9219cae8f2b7d49, d961985de7874e1990a19bb9b2da14c2; no voters: 
I20240502 21:26:49.630048  3254 raft_consensus.cc:2781] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:49.630227  3254 raft_consensus.cc:686] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [term 1 LEADER]: Becoming Leader. State: Replica: c616208720db4b14a9219cae8f2b7d49, State: Running, Role: LEADER
I20240502 21:26:49.630582  3254 consensus_queue.cc:237] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:49.633473  2047 catalog_manager.cc:5484] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 reported cstate change: term changed from 0 to 1, leader changed from <none> to c616208720db4b14a9219cae8f2b7d49 (127.27.196.67). New cstate: current_term: 1 leader_uuid: "c616208720db4b14a9219cae8f2b7d49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } }
I20240502 21:26:49.681591  3384 raft_consensus.cc:482] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:49.681818  3384 raft_consensus.cc:504] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:49.682535  3384 leader_election.cc:288] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025)
I20240502 21:26:49.683523  2194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be2cbcd79c1e40f49701271bb247f4d0" candidate_uuid: "68f426685b4c4c90bda61ef52177cc22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d961985de7874e1990a19bb9b2da14c2" is_pre_election: true
I20240502 21:26:49.683789  2194 raft_consensus.cc:2454] T be2cbcd79c1e40f49701271bb247f4d0 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68f426685b4c4c90bda61ef52177cc22 in term 0.
I20240502 21:26:49.684367  2264 leader_election.cc:302] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 68f426685b4c4c90bda61ef52177cc22, d961985de7874e1990a19bb9b2da14c2; no voters: 
I20240502 21:26:49.684687  3384 raft_consensus.cc:2781] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:49.684793  3384 raft_consensus.cc:482] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:49.684890  3384 raft_consensus.cc:3037] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:49.686949  3384 raft_consensus.cc:504] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:49.687547  3384 leader_election.cc:288] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 1 election: Requested vote from peers d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025)
I20240502 21:26:49.688585  2194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be2cbcd79c1e40f49701271bb247f4d0" candidate_uuid: "68f426685b4c4c90bda61ef52177cc22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d961985de7874e1990a19bb9b2da14c2"
I20240502 21:26:49.689023  2194 raft_consensus.cc:3037] T be2cbcd79c1e40f49701271bb247f4d0 P d961985de7874e1990a19bb9b2da14c2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:49.691047  2194 raft_consensus.cc:2454] T be2cbcd79c1e40f49701271bb247f4d0 P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68f426685b4c4c90bda61ef52177cc22 in term 1.
I20240502 21:26:49.691692  2264 leader_election.cc:302] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 68f426685b4c4c90bda61ef52177cc22, d961985de7874e1990a19bb9b2da14c2; no voters: 
I20240502 21:26:49.692102  3384 raft_consensus.cc:2781] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:49.692287  3384 raft_consensus.cc:686] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Becoming Leader. State: Replica: 68f426685b4c4c90bda61ef52177cc22, State: Running, Role: LEADER
I20240502 21:26:49.692615  3384 consensus_queue.cc:237] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } }
I20240502 21:26:49.695528  2047 catalog_manager.cc:5484] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "68f426685b4c4c90bda61ef52177cc22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } }
I20240502 21:26:50.065519  3254 consensus_queue.cc:1035] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:50.130436  3413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:50.130859  3413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:50.140452  3413 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:26:50.143180  3409 consensus_queue.cc:1035] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240502 21:26:50.247484  3384 consensus_queue.cc:1035] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 193.815 ms
  time per row: 0.0969074 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240502 21:26:51.463726 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.68:44575
--local_ip_for_outbound_sockets=127.27.196.68
--tserver_master_addrs=127.27.196.126:43629
--webserver_port=40277
--webserver_interface=127.27.196.68
--builtin_ntp_servers=127.27.196.84:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 21:26:51.858963  3466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:51.859341  3466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:51.859452  3466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:51.867538  3466 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:51.867703  3466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:51.867887  3466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.68
I20240502 21:26:51.878193  3466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.68:44575
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.27.196.68
--webserver_port=40277
--tserver_master_addrs=127.27.196.126:43629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.68
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:51.879774  3466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:51.881810  3466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:51.897496  3474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:51.897541  3472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:51.898356  3471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:51.899178  3466 server_base.cc:1034] running on GCE node
I20240502 21:26:51.900274  3466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:51.901818  3466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:51.903210  3466 hybrid_clock.cc:648] HybridClock initialized: now 1714685211903089 us; error 55 us; skew 500 ppm
I20240502 21:26:51.903678  3466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:51.907691  3466 webserver.cc:460] Webserver started at http://127.27.196.68:40277/ using document root <none> and password file <none>
I20240502 21:26:51.908845  3466 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:51.909054  3466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:51.916612  3466 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.003s	sys 0.001s
I20240502 21:26:51.920325  3480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:51.921919  3466 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240502 21:26:51.922138  3466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "13451be0f7134a2c89c0c9330177213e"
format_stamp: "Formatted at 2024-05-02 21:26:37 on dist-test-slave-3bs1"
I20240502 21:26:51.923182  3466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:51.944680  3466 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:26:51.945648  3466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:51.945972  3466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:51.947243  3466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:51.949790  3487 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240502 21:26:51.956104  3466 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 21:26:51.956247  3466 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
I20240502 21:26:51.956733  3466 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 21:26:51.960911  3466 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 21:26:51.961077  3466 ts_tablet_manager.cc:569] Time spent register tablets: real 0.004s	user 0.003s	sys 0.000s
I20240502 21:26:51.961797  3487 tablet_bootstrap.cc:492] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e: Bootstrap starting.
I20240502 21:26:51.982894  3487 log.cc:824] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:51.987830  3487 tablet_bootstrap.cc:492] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 21:26:51.989181  3487 tablet_bootstrap.cc:492] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e: Bootstrap complete.
I20240502 21:26:51.990495  3487 ts_tablet_manager.cc:1367] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e: Time spent bootstrapping tablet: real 0.029s	user 0.020s	sys 0.008s
I20240502 21:26:52.003924  3487 raft_consensus.cc:348] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:52.005447  3487 raft_consensus.cc:729] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13451be0f7134a2c89c0c9330177213e, State: Initialized, Role: FOLLOWER
I20240502 21:26:52.006757  3487 consensus_queue.cc:260] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:52.009158  3487 ts_tablet_manager.cc:1398] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e: Time spent starting tablet: real 0.019s	user 0.015s	sys 0.004s
I20240502 21:26:52.020403  3466 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.68:44575
I20240502 21:26:52.020505  3598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.68:44575 every 8 connection(s)
I20240502 21:26:52.022271  3466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
W20240502 21:26:52.025230  3600 tablet.cc:2369] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:52.032338 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 3466
I20240502 21:26:52.033205 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.69:37783
--local_ip_for_outbound_sockets=127.27.196.69
--tserver_master_addrs=127.27.196.126:43629
--webserver_port=38529
--webserver_interface=127.27.196.69
--builtin_ntp_servers=127.27.196.84:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:26:52.040109  3599 heartbeater.cc:347] Connected to a master server at 127.27.196.126:43629
I20240502 21:26:52.040462  3599 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:52.041306  3599 heartbeater.cc:511] Master 127.27.196.126:43629 requested a full tablet report, sending...
I20240502 21:26:52.044131  2050 ts_manager.cc:194] Re-registered known tserver with Master: 13451be0f7134a2c89c0c9330177213e (127.27.196.68:44575)
I20240502 21:26:52.045928  2050 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.68:37645
W20240502 21:26:52.401715  3603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:52.402069  3603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:52.402164  3603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:52.410140  3603 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:52.410290  3603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:52.410409  3603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.69
I20240502 21:26:52.420045  3603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.69:37783
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.27.196.69
--webserver_port=38529
--tserver_master_addrs=127.27.196.126:43629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.69
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:52.421485  3603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:52.423398  3603 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:52.438397  3609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:52.438985  3611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:52.439561  3608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:52.440411  3603 server_base.cc:1034] running on GCE node
I20240502 21:26:52.441128  3603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:52.442304  3603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:52.443632  3603 hybrid_clock.cc:648] HybridClock initialized: now 1714685212443556 us; error 59 us; skew 500 ppm
I20240502 21:26:52.444164  3603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:52.447630  3603 webserver.cc:460] Webserver started at http://127.27.196.69:38529/ using document root <none> and password file <none>
I20240502 21:26:52.448756  3603 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:52.448930  3603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:52.456681  3603 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.003s	sys 0.000s
I20240502 21:26:52.460209  3617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:52.462347  3603 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
I20240502 21:26:52.462666  3603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "4e8c16d1a1484a41a54b5ddc198becd3"
format_stamp: "Formatted at 2024-05-02 21:26:37 on dist-test-slave-3bs1"
I20240502 21:26:52.463893  3603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:52.497880  3603 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:26:52.499055  3603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:52.499469  3603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:52.500998  3603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:52.503151  3603 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:52.503248  3603 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.001s
I20240502 21:26:52.503403  3603 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:52.503459  3603 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:52.547071  3603 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.69:37783
I20240502 21:26:52.547199  3731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.69:37783 every 8 connection(s)
I20240502 21:26:52.548836  3603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:26:52.551303 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 3603
I20240502 21:26:52.552075 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.70:46179
--local_ip_for_outbound_sockets=127.27.196.70
--tserver_master_addrs=127.27.196.126:43629
--webserver_port=36905
--webserver_interface=127.27.196.70
--builtin_ntp_servers=127.27.196.84:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:26:52.571978  3732 heartbeater.cc:347] Connected to a master server at 127.27.196.126:43629
I20240502 21:26:52.572448  3732 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:52.573652  3732 heartbeater.cc:511] Master 127.27.196.126:43629 requested a full tablet report, sending...
I20240502 21:26:52.575742  2050 ts_manager.cc:194] Re-registered known tserver with Master: 4e8c16d1a1484a41a54b5ddc198becd3 (127.27.196.69:37783)
I20240502 21:26:52.576653  2050 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.69:37615
W20240502 21:26:52.958992  3736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:52.959450  3736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:52.959595  3736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:52.967823  3736 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:52.968046  3736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:52.968219  3736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.70
I20240502 21:26:52.980249  3736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.70:46179
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.27.196.70
--webserver_port=36905
--tserver_master_addrs=127.27.196.126:43629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.70
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:52.982136  3736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:52.984686  3736 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:52.999725  3744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:53.000067  3741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:53.000231  3742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:53.001061  3736 server_base.cc:1034] running on GCE node
I20240502 21:26:53.001754  3736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:53.002789  3736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:53.004035  3736 hybrid_clock.cc:648] HybridClock initialized: now 1714685213003982 us; error 49 us; skew 500 ppm
I20240502 21:26:53.004420  3736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:53.007233  3736 webserver.cc:460] Webserver started at http://127.27.196.70:36905/ using document root <none> and password file <none>
I20240502 21:26:53.008136  3736 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:53.008278  3736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:53.015369  3736 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.007s	sys 0.000s
I20240502 21:26:53.018735  3750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:53.020488  3736 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 21:26:53.020716  3736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "1193f1d0d2c843139d98458cf4035eb8"
format_stamp: "Formatted at 2024-05-02 21:26:38 on dist-test-slave-3bs1"
I20240502 21:26:53.021679  3736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:53.044498  3736 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:26:53.045783  3736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:53.046213  3736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:53.047679  3736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:53.049139  3599 heartbeater.cc:503] Master 127.27.196.126:43629 was elected leader, sending a full tablet report...
I20240502 21:26:53.050029  3736 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:53.050249  3736 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.001s	sys 0.000s
I20240502 21:26:53.050451  3736 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:53.050550  3736 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:53.094342  3736 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.70:46179
I20240502 21:26:53.094373  3866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.70:46179 every 8 connection(s)
I20240502 21:26:53.096102  3736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 21:26:53.102424 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 3736
I20240502 21:26:53.103255 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.71:39583
--local_ip_for_outbound_sockets=127.27.196.71
--tserver_master_addrs=127.27.196.126:43629
--webserver_port=43309
--webserver_interface=127.27.196.71
--builtin_ntp_servers=127.27.196.84:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:26:53.113516  3867 heartbeater.cc:347] Connected to a master server at 127.27.196.126:43629
I20240502 21:26:53.113883  3867 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:53.114740  3867 heartbeater.cc:511] Master 127.27.196.126:43629 requested a full tablet report, sending...
I20240502 21:26:53.116801  2050 ts_manager.cc:194] Re-registered known tserver with Master: 1193f1d0d2c843139d98458cf4035eb8 (127.27.196.70:46179)
I20240502 21:26:53.118074  2050 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.70:60891
I20240502 21:26:53.366614  3872 raft_consensus.cc:482] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:53.367038  3872 raft_consensus.cc:504] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:53.368937  3872 leader_election.cc:288] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253)
I20240502 21:26:53.377421  2330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae512d4d5874a909b1f4186c4b9785a" candidate_uuid: "13451be0f7134a2c89c0c9330177213e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "68f426685b4c4c90bda61ef52177cc22" is_pre_election: true
I20240502 21:26:53.377717  2330 raft_consensus.cc:2454] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13451be0f7134a2c89c0c9330177213e in term 1.
I20240502 21:26:53.378432  3481 leader_election.cc:302] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 13451be0f7134a2c89c0c9330177213e, 68f426685b4c4c90bda61ef52177cc22; no voters: 
I20240502 21:26:53.378965  3872 raft_consensus.cc:2781] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:26:53.379148  3872 raft_consensus.cc:482] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:53.379273  3872 raft_consensus.cc:3037] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:53.383427  3872 raft_consensus.cc:504] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:53.384145  3872 leader_election.cc:288] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [CANDIDATE]: Term 2 election: Requested vote from peers 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253)
I20240502 21:26:53.384873  2330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae512d4d5874a909b1f4186c4b9785a" candidate_uuid: "13451be0f7134a2c89c0c9330177213e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "68f426685b4c4c90bda61ef52177cc22"
I20240502 21:26:53.385087  2330 raft_consensus.cc:3037] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:53.387230  2330 raft_consensus.cc:2454] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13451be0f7134a2c89c0c9330177213e in term 2.
I20240502 21:26:53.387825  3481 leader_election.cc:302] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 13451be0f7134a2c89c0c9330177213e, 68f426685b4c4c90bda61ef52177cc22; no voters: 
I20240502 21:26:53.388231  3872 raft_consensus.cc:2781] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:26:53.388686  3872 raft_consensus.cc:686] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [term 2 LEADER]: Becoming Leader. State: Replica: 13451be0f7134a2c89c0c9330177213e, State: Running, Role: LEADER
I20240502 21:26:53.389250  3872 consensus_queue.cc:237] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } }
I20240502 21:26:53.394048  2050 catalog_manager.cc:5484] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "13451be0f7134a2c89c0c9330177213e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } }
W20240502 21:26:53.477284  3871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:53.477619  3871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:53.477730  3871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:53.486537  3871 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:26:53.486682  3871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:53.486861  3871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.71
I20240502 21:26:53.497402  3871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.71:39583
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.27.196.71
--webserver_port=43309
--tserver_master_addrs=127.27.196.126:43629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.71
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:53.498971  3871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:53.500909  3871 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:53.515064  3885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:53.516430  3888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:53.517227  3886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:53.518151  3871 server_base.cc:1034] running on GCE node
I20240502 21:26:53.518831  3871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:53.520136  3871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:53.521389  3871 hybrid_clock.cc:648] HybridClock initialized: now 1714685213521337 us; error 42 us; skew 500 ppm
I20240502 21:26:53.521761  3871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:53.524519  3871 webserver.cc:460] Webserver started at http://127.27.196.71:43309/ using document root <none> and password file <none>
I20240502 21:26:53.525382  3871 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:53.525518  3871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:53.532459  3871 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.003s	sys 0.000s
I20240502 21:26:53.537351  3894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:53.539919  3871 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.006s	sys 0.000s
I20240502 21:26:53.540174  3871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "d306b26127e04e8c85a98edf1b41978a"
format_stamp: "Formatted at 2024-05-02 21:26:38 on dist-test-slave-3bs1"
I20240502 21:26:53.541146  3871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:53.574582  3871 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:26:53.575663  3871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:53.575978  3871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:53.577215  3871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:53.579099  3871 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:53.579212  3871 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.001s	sys 0.000s
I20240502 21:26:53.579355  3871 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:53.579424  3871 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:26:53.580132  3732 heartbeater.cc:503] Master 127.27.196.126:43629 was elected leader, sending a full tablet report...
I20240502 21:26:53.623920  3871 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.71:39583
I20240502 21:26:53.623986  4010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.71:39583 every 8 connection(s)
I20240502 21:26:53.625586  3871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 21:26:53.631073 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 3871
I20240502 21:26:53.631481 28433 test_util.cc:268] Using random seed: 1125082064
I20240502 21:26:53.642532  4011 heartbeater.cc:347] Connected to a master server at 127.27.196.126:43629
I20240502 21:26:53.642940  4011 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:53.643875  4011 heartbeater.cc:511] Master 127.27.196.126:43629 requested a full tablet report, sending...
I20240502 21:26:53.645414  2050 ts_manager.cc:194] Re-registered known tserver with Master: d306b26127e04e8c85a98edf1b41978a (127.27.196.71:39583)
I20240502 21:26:53.646293  2050 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.71:40753
I20240502 21:26:53.652242 28433 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240502 21:26:53.652823 28433 test_util.cc:268] Using random seed: 1125103408
I20240502 21:26:53.670029 28433 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240502 21:26:53.670806 28433 test_util.cc:268] Using random seed: 1125121315
I20240502 21:26:53.698370 28433 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240502 21:26:53.706010 28433 test_util.cc:268] Using random seed: 1125156528
I20240502 21:26:53.762217 28433 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240502 21:26:53.765851 28433 test_util.cc:268] Using random seed: 1125216400
W20240502 21:26:53.821861  4060 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.27.196.126:43629, user_credentials={real_user=slave}} blocked reactor thread for 45264us
I20240502 21:26:53.870204 28433 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240502 21:26:53.936965  2329 raft_consensus.cc:1261] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [term 2 FOLLOWER]: Refusing update from remote peer 13451be0f7134a2c89c0c9330177213e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:26:53.957324  4096 consensus_queue.cc:1035] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [LEADER]: Connected to new peer: Peer: permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:53.986224  4072 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.27.196.126:43629, user_credentials={real_user=slave}} blocked reactor thread for 75842.1us
I20240502 21:26:54.120975  3867 heartbeater.cc:503] Master 127.27.196.126:43629 was elected leader, sending a full tablet report...
I20240502 21:26:54.649643  4011 heartbeater.cc:503] Master 127.27.196.126:43629 was elected leader, sending a full tablet report...
I20240502 21:26:55.471050  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:55.472060  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:55.473538  3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:55.498996  2308 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:26:55.615833  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:55.716091  3665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:55.876569  2174 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 21:26:57.049894  2398 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.66:36253, user_credentials={real_user=slave}} blocked reactor thread for 142421us
I20240502 21:26:57.064978  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:57.068434  2308 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:26:57.074272  3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:57.074776  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:57.102864  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:57.141571  3665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:57.169051  2174 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:26:57.539330  2330 consensus_queue.cc:237] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 1.49, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } }
I20240502 21:26:57.549388  2194 raft_consensus.cc:1261] T d7db9b73c0594885b383b059520414e9 P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Refusing update from remote peer 68f426685b4c4c90bda61ef52177cc22: Log matching property violated. Preceding OpId in replica: term: 1 index: 49. Preceding OpId from leader: term: 1 index: 50. (index mismatch)
I20240502 21:26:57.551481  4139 consensus_queue.cc:1035] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20240502 21:26:57.570784  2194 raft_consensus.cc:2932] T d7db9b73c0594885b383b059520414e9 P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Committing config change with OpId 1.50: config changed from index -1 to 50, VOTER 1193f1d0d2c843139d98458cf4035eb8 (127.27.196.70) added. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } }
I20240502 21:26:57.572288  4139 raft_consensus.cc:2932] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Committing config change with OpId 1.50: config changed from index -1 to 50, VOTER 1193f1d0d2c843139d98458cf4035eb8 (127.27.196.70) added. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } }
I20240502 21:26:57.680243  2045 catalog_manager.cc:5484] T d7db9b73c0594885b383b059520414e9 P d961985de7874e1990a19bb9b2da14c2 reported cstate change: config changed from index -1 to 50, VOTER 1193f1d0d2c843139d98458cf4035eb8 (127.27.196.70) added. New cstate: current_term: 1 leader_uuid: "68f426685b4c4c90bda61ef52177cc22" committed_config { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } }
W20240502 21:26:57.717346  2265 consensus_peers.cc:485] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22 -> Peer 1193f1d0d2c843139d98458cf4035eb8 (127.27.196.70:46179): Couldn't send request to peer 1193f1d0d2c843139d98458cf4035eb8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7db9b73c0594885b383b059520414e9. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:57.978147  2467 consensus_queue.cc:237] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 1.64, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
I20240502 21:26:57.983248  2329 raft_consensus.cc:1261] T d64bd5e7c22749078d82ad8da12d75dd P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Refusing update from remote peer c616208720db4b14a9219cae8f2b7d49: Log matching property violated. Preceding OpId in replica: term: 1 index: 64. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240502 21:26:57.984048  3340 consensus_queue.cc:1035] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20240502 21:26:57.994282  4110 raft_consensus.cc:2932] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49 [term 1 LEADER]: Committing config change with OpId 1.65: config changed from index -1 to 65, VOTER d306b26127e04e8c85a98edf1b41978a (127.27.196.71) added. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:26:58.002259  2050 catalog_manager.cc:5484] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49 reported cstate change: config changed from index -1 to 65, VOTER d306b26127e04e8c85a98edf1b41978a (127.27.196.71) added. New cstate: current_term: 1 leader_uuid: "c616208720db4b14a9219cae8f2b7d49" committed_config { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
W20240502 21:26:58.013545  2398 consensus_peers.cc:485] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49 -> Peer d306b26127e04e8c85a98edf1b41978a (127.27.196.71:39583): Couldn't send request to peer d306b26127e04e8c85a98edf1b41978a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d64bd5e7c22749078d82ad8da12d75dd. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:58.017572  2329 raft_consensus.cc:2932] T d64bd5e7c22749078d82ad8da12d75dd P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index -1 to 65, VOTER d306b26127e04e8c85a98edf1b41978a (127.27.196.71) added. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:26:58.103068  4292 ts_tablet_manager.cc:902] T d7db9b73c0594885b383b059520414e9 P 1193f1d0d2c843139d98458cf4035eb8: Initiating tablet copy from peer 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253)
I20240502 21:26:58.208590  4292 tablet_copy_client.cc:308] T d7db9b73c0594885b383b059520414e9 P 1193f1d0d2c843139d98458cf4035eb8: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:36253
I20240502 21:26:58.229662  2353 tablet_copy_service.cc:140] P 68f426685b4c4c90bda61ef52177cc22: Received BeginTabletCopySession request for tablet d7db9b73c0594885b383b059520414e9 from peer 1193f1d0d2c843139d98458cf4035eb8 ({username='slave'} at 127.27.196.70:56283)
I20240502 21:26:58.230108  2353 tablet_copy_service.cc:161] P 68f426685b4c4c90bda61ef52177cc22: Beginning new tablet copy session on tablet d7db9b73c0594885b383b059520414e9 from peer 1193f1d0d2c843139d98458cf4035eb8 at {username='slave'} at 127.27.196.70:56283: session id = 1193f1d0d2c843139d98458cf4035eb8-d7db9b73c0594885b383b059520414e9
I20240502 21:26:58.248755  2353 tablet_copy_source_session.cc:205] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:58.256141  4292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7db9b73c0594885b383b059520414e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.321002  4292 tablet_copy_client.cc:783] T d7db9b73c0594885b383b059520414e9 P 1193f1d0d2c843139d98458cf4035eb8: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:58.321583  4292 tablet_copy_client.cc:647] T d7db9b73c0594885b383b059520414e9 P 1193f1d0d2c843139d98458cf4035eb8: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:58.328092  2329 consensus_queue.cc:237] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 1.81, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } }
I20240502 21:26:58.329324  4292 tablet_copy_client.cc:523] T d7db9b73c0594885b383b059520414e9 P 1193f1d0d2c843139d98458cf4035eb8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:58.339044  4292 tablet_bootstrap.cc:492] T d7db9b73c0594885b383b059520414e9 P 1193f1d0d2c843139d98458cf4035eb8: Bootstrap starting.
I20240502 21:26:58.352391  2464 raft_consensus.cc:1261] T 2779949906ff486da691638de7374050 P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Refusing update from remote peer 68f426685b4c4c90bda61ef52177cc22: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20240502 21:26:58.361521  4139 consensus_queue.cc:1035] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20240502 21:26:58.365320  4292 log.cc:824] T d7db9b73c0594885b383b059520414e9 P 1193f1d0d2c843139d98458cf4035eb8: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:58.375207  4139 raft_consensus.cc:2932] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Committing config change with OpId 1.82: config changed from index -1 to 82, VOTER 4e8c16d1a1484a41a54b5ddc198becd3 (127.27.196.69) added. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } }
I20240502 21:26:58.382638  2463 raft_consensus.cc:2932] T 2779949906ff486da691638de7374050 P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index -1 to 82, VOTER 4e8c16d1a1484a41a54b5ddc198becd3 (127.27.196.69) added. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } }
I20240502 21:26:58.403553  2050 catalog_manager.cc:5484] T 2779949906ff486da691638de7374050 P c616208720db4b14a9219cae8f2b7d49 reported cstate change: config changed from index -1 to 82, VOTER 4e8c16d1a1484a41a54b5ddc198becd3 (127.27.196.69) added. New cstate: current_term: 1 leader_uuid: "68f426685b4c4c90bda61ef52177cc22" committed_config { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } }
W20240502 21:26:58.443535  2265 consensus_peers.cc:485] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 -> Peer 4e8c16d1a1484a41a54b5ddc198becd3 (127.27.196.69:37783): Couldn't send request to peer 4e8c16d1a1484a41a54b5ddc198becd3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2779949906ff486da691638de7374050. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:58.615783  4330 ts_tablet_manager.cc:902] T d64bd5e7c22749078d82ad8da12d75dd P d306b26127e04e8c85a98edf1b41978a: Initiating tablet copy from peer c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687)
I20240502 21:26:58.682288  4330 tablet_copy_client.cc:308] T d64bd5e7c22749078d82ad8da12d75dd P d306b26127e04e8c85a98edf1b41978a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:46687
I20240502 21:26:58.751324  2487 tablet_copy_service.cc:140] P c616208720db4b14a9219cae8f2b7d49: Received BeginTabletCopySession request for tablet d64bd5e7c22749078d82ad8da12d75dd from peer d306b26127e04e8c85a98edf1b41978a ({username='slave'} at 127.27.196.71:42883)
I20240502 21:26:58.751677  2487 tablet_copy_service.cc:161] P c616208720db4b14a9219cae8f2b7d49: Beginning new tablet copy session on tablet d64bd5e7c22749078d82ad8da12d75dd from peer d306b26127e04e8c85a98edf1b41978a at {username='slave'} at 127.27.196.71:42883: session id = d306b26127e04e8c85a98edf1b41978a-d64bd5e7c22749078d82ad8da12d75dd
I20240502 21:26:58.755985  2487 tablet_copy_source_session.cc:205] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:58.763846  4330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d64bd5e7c22749078d82ad8da12d75dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.766335  4292 tablet_bootstrap.cc:492] T d7db9b73c0594885b383b059520414e9 P 1193f1d0d2c843139d98458cf4035eb8: Bootstrap replayed 1/1 log segments. Stats: ops{read=63 overwritten=0 applied=63 ignored=0} inserts{seen=735 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:58.770524  4292 tablet_bootstrap.cc:492] T d7db9b73c0594885b383b059520414e9 P 1193f1d0d2c843139d98458cf4035eb8: Bootstrap complete.
I20240502 21:26:58.774674  4292 ts_tablet_manager.cc:1367] T d7db9b73c0594885b383b059520414e9 P 1193f1d0d2c843139d98458cf4035eb8: Time spent bootstrapping tablet: real 0.436s	user 0.148s	sys 0.063s
I20240502 21:26:58.783932  4292 raft_consensus.cc:348] T d7db9b73c0594885b383b059520414e9 P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } }
I20240502 21:26:58.784847  4292 raft_consensus.cc:729] T d7db9b73c0594885b383b059520414e9 P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1193f1d0d2c843139d98458cf4035eb8, State: Initialized, Role: FOLLOWER
I20240502 21:26:58.786001  4292 consensus_queue.cc:260] T d7db9b73c0594885b383b059520414e9 P 1193f1d0d2c843139d98458cf4035eb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 63, Last appended: 1.63, Last appended by leader: 63, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } }
I20240502 21:26:58.802011  4330 tablet_copy_client.cc:783] T d64bd5e7c22749078d82ad8da12d75dd P d306b26127e04e8c85a98edf1b41978a: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:58.806571  4330 tablet_copy_client.cc:647] T d64bd5e7c22749078d82ad8da12d75dd P d306b26127e04e8c85a98edf1b41978a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:58.812795  4330 tablet_copy_client.cc:523] T d64bd5e7c22749078d82ad8da12d75dd P d306b26127e04e8c85a98edf1b41978a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 21:26:58.813616  2263 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.67:46687, user_credentials={real_user=slave}} blocked reactor thread for 68558.4us
I20240502 21:26:58.816838  4292 ts_tablet_manager.cc:1398] T d7db9b73c0594885b383b059520414e9 P 1193f1d0d2c843139d98458cf4035eb8: Time spent starting tablet: real 0.042s	user 0.009s	sys 0.014s
I20240502 21:26:58.818017  4330 tablet_bootstrap.cc:492] T d64bd5e7c22749078d82ad8da12d75dd P d306b26127e04e8c85a98edf1b41978a: Bootstrap starting.
I20240502 21:26:58.819900  2353 tablet_copy_service.cc:342] P 68f426685b4c4c90bda61ef52177cc22: Request end of tablet copy session 1193f1d0d2c843139d98458cf4035eb8-d7db9b73c0594885b383b059520414e9 received from {username='slave'} at 127.27.196.70:56283
I20240502 21:26:58.820139  2353 tablet_copy_service.cc:433] P 68f426685b4c4c90bda61ef52177cc22: ending tablet copy session 1193f1d0d2c843139d98458cf4035eb8-d7db9b73c0594885b383b059520414e9 on tablet d7db9b73c0594885b383b059520414e9 with peer 1193f1d0d2c843139d98458cf4035eb8
I20240502 21:26:58.864591  4330 log.cc:824] T d64bd5e7c22749078d82ad8da12d75dd P d306b26127e04e8c85a98edf1b41978a: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:58.874888  4342 ts_tablet_manager.cc:902] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3: Initiating tablet copy from peer 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253)
I20240502 21:26:58.902171  4342 tablet_copy_client.cc:308] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:36253
I20240502 21:26:58.955168  2195 consensus_queue.cc:237] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 1.80, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } }
I20240502 21:26:58.970274  2353 tablet_copy_service.cc:140] P 68f426685b4c4c90bda61ef52177cc22: Received BeginTabletCopySession request for tablet 2779949906ff486da691638de7374050 from peer 4e8c16d1a1484a41a54b5ddc198becd3 ({username='slave'} at 127.27.196.69:44867)
I20240502 21:26:58.970536  2353 tablet_copy_service.cc:161] P 68f426685b4c4c90bda61ef52177cc22: Beginning new tablet copy session on tablet 2779949906ff486da691638de7374050 from peer 4e8c16d1a1484a41a54b5ddc198becd3 at {username='slave'} at 127.27.196.69:44867: session id = 4e8c16d1a1484a41a54b5ddc198becd3-2779949906ff486da691638de7374050
I20240502 21:26:58.973469  2353 tablet_copy_source_session.cc:205] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:58.975658  2464 raft_consensus.cc:1261] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Refusing update from remote peer d961985de7874e1990a19bb9b2da14c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 80. Preceding OpId from leader: term: 1 index: 81. (index mismatch)
I20240502 21:26:58.978979  4114 consensus_queue.cc:1035] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.001s
I20240502 21:26:58.984699  4114 raft_consensus.cc:2932] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [term 1 LEADER]: Committing config change with OpId 1.81: config changed from index -1 to 81, VOTER 13451be0f7134a2c89c0c9330177213e (127.27.196.68) added. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } }
I20240502 21:26:58.986178  4342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2779949906ff486da691638de7374050. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.986097  2464 raft_consensus.cc:2932] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Committing config change with OpId 1.81: config changed from index -1 to 81, VOTER 13451be0f7134a2c89c0c9330177213e (127.27.196.68) added. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } }
I20240502 21:26:59.000133  2050 catalog_manager.cc:5484] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 reported cstate change: config changed from index -1 to 81, VOTER 13451be0f7134a2c89c0c9330177213e (127.27.196.68) added. New cstate: current_term: 1 leader_uuid: "d961985de7874e1990a19bb9b2da14c2" committed_config { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } }
I20240502 21:26:59.088842  4342 tablet_copy_client.cc:783] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:59.098820  4342 tablet_copy_client.cc:647] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:59.107383  4342 tablet_copy_client.cc:523] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:59.112646  4342 tablet_bootstrap.cc:492] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3: Bootstrap starting.
W20240502 21:26:59.117524  2130 consensus_peers.cc:485] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 -> Peer 13451be0f7134a2c89c0c9330177213e (127.27.196.68:44575): Couldn't send request to peer 13451be0f7134a2c89c0c9330177213e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e1c5a5cc57243d7bf6e212e3a32dfe3. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:59.175031  3818 raft_consensus.cc:1203] T d7db9b73c0594885b383b059520414e9 P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.49->[1.50-1.75]   Dedup: 1.63->[1.64-1.75]
I20240502 21:26:59.213214  4342 log.cc:824] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:59.273573  2329 consensus_queue.cc:237] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 1.65, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } }
W20240502 21:26:59.276530  2265 consensus_peers.cc:485] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22 -> Peer 1193f1d0d2c843139d98458cf4035eb8 (127.27.196.70:46179): Couldn't send request to peer 1193f1d0d2c843139d98458cf4035eb8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a38d2b4246e497fa91d2a0a8b127244. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:59.289709  2195 raft_consensus.cc:1261] T 6a38d2b4246e497fa91d2a0a8b127244 P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Refusing update from remote peer 68f426685b4c4c90bda61ef52177cc22: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240502 21:26:59.291810  4330 tablet_bootstrap.cc:492] T d64bd5e7c22749078d82ad8da12d75dd P d306b26127e04e8c85a98edf1b41978a: Bootstrap replayed 1/1 log segments. Stats: ops{read=75 overwritten=0 applied=74 ignored=0} inserts{seen=705 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 21:26:59.292690  4330 tablet_bootstrap.cc:492] T d64bd5e7c22749078d82ad8da12d75dd P d306b26127e04e8c85a98edf1b41978a: Bootstrap complete.
I20240502 21:26:59.294309  4330 ts_tablet_manager.cc:1367] T d64bd5e7c22749078d82ad8da12d75dd P d306b26127e04e8c85a98edf1b41978a: Time spent bootstrapping tablet: real 0.477s	user 0.195s	sys 0.001s
I20240502 21:26:59.302531  3442 consensus_queue.cc:1035] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20240502 21:26:59.308182  4330 raft_consensus.cc:348] T d64bd5e7c22749078d82ad8da12d75dd P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
I20240502 21:26:59.324319  4217 raft_consensus.cc:2932] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Committing config change with OpId 1.66: config changed from index -1 to 66, VOTER 1193f1d0d2c843139d98458cf4035eb8 (127.27.196.70) added. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } }
I20240502 21:26:59.327355  4330 raft_consensus.cc:729] T d64bd5e7c22749078d82ad8da12d75dd P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d306b26127e04e8c85a98edf1b41978a, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.328781  4330 consensus_queue.cc:260] T d64bd5e7c22749078d82ad8da12d75dd P d306b26127e04e8c85a98edf1b41978a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 74, Last appended: 1.75, Last appended by leader: 75, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
I20240502 21:26:59.339294  2045 catalog_manager.cc:5484] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22 reported cstate change: config changed from index -1 to 66, VOTER 1193f1d0d2c843139d98458cf4035eb8 (127.27.196.70) added. New cstate: current_term: 1 leader_uuid: "68f426685b4c4c90bda61ef52177cc22" committed_config { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } }
I20240502 21:26:59.349033  2194 raft_consensus.cc:2932] T 6a38d2b4246e497fa91d2a0a8b127244 P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index -1 to 66, VOTER 1193f1d0d2c843139d98458cf4035eb8 (127.27.196.70) added. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } }
I20240502 21:26:59.360656  4330 ts_tablet_manager.cc:1398] T d64bd5e7c22749078d82ad8da12d75dd P d306b26127e04e8c85a98edf1b41978a: Time spent starting tablet: real 0.066s	user 0.029s	sys 0.004s
I20240502 21:26:59.370755  2487 tablet_copy_service.cc:342] P c616208720db4b14a9219cae8f2b7d49: Request end of tablet copy session d306b26127e04e8c85a98edf1b41978a-d64bd5e7c22749078d82ad8da12d75dd received from {username='slave'} at 127.27.196.71:42883
I20240502 21:26:59.370993  2487 tablet_copy_service.cc:433] P c616208720db4b14a9219cae8f2b7d49: ending tablet copy session d306b26127e04e8c85a98edf1b41978a-d64bd5e7c22749078d82ad8da12d75dd on tablet d64bd5e7c22749078d82ad8da12d75dd with peer d306b26127e04e8c85a98edf1b41978a
I20240502 21:26:59.612336  4398 ts_tablet_manager.cc:902] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e: Initiating tablet copy from peer d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025)
I20240502 21:26:59.629945  4398 tablet_copy_client.cc:308] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:37025
I20240502 21:26:59.637634  4342 tablet_bootstrap.cc:492] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=88 overwritten=0 applied=88 ignored=0} inserts{seen=773 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:59.650159  4342 tablet_bootstrap.cc:492] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3: Bootstrap complete.
I20240502 21:26:59.653111  4342 ts_tablet_manager.cc:1367] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3: Time spent bootstrapping tablet: real 0.541s	user 0.227s	sys 0.007s
I20240502 21:26:59.680370  4342 raft_consensus.cc:348] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } }
I20240502 21:26:59.681106  4342 raft_consensus.cc:729] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e8c16d1a1484a41a54b5ddc198becd3, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.685520  4342 consensus_queue.cc:260] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 88, Last appended: 1.88, Last appended by leader: 88, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } }
I20240502 21:26:59.704007  2215 tablet_copy_service.cc:140] P d961985de7874e1990a19bb9b2da14c2: Received BeginTabletCopySession request for tablet 6e1c5a5cc57243d7bf6e212e3a32dfe3 from peer 13451be0f7134a2c89c0c9330177213e ({username='slave'} at 127.27.196.68:52125)
I20240502 21:26:59.704404  2215 tablet_copy_service.cc:161] P d961985de7874e1990a19bb9b2da14c2: Beginning new tablet copy session on tablet 6e1c5a5cc57243d7bf6e212e3a32dfe3 from peer 13451be0f7134a2c89c0c9330177213e at {username='slave'} at 127.27.196.68:52125: session id = 13451be0f7134a2c89c0c9330177213e-6e1c5a5cc57243d7bf6e212e3a32dfe3
I20240502 21:26:59.708693  2215 tablet_copy_source_session.cc:205] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:59.713284  4398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e1c5a5cc57243d7bf6e212e3a32dfe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.724815  4398 tablet_copy_client.cc:783] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:59.727017  4342 ts_tablet_manager.cc:1398] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3: Time spent starting tablet: real 0.072s	user 0.013s	sys 0.003s
I20240502 21:26:59.727913  4398 tablet_copy_client.cc:647] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:59.731474  4398 tablet_copy_client.cc:523] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:59.735525  4398 tablet_bootstrap.cc:492] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e: Bootstrap starting.
I20240502 21:26:59.735509  3966 raft_consensus.cc:1203] T d64bd5e7c22749078d82ad8da12d75dd P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.64->[1.65-1.83]   Dedup: 1.75->[1.76-1.83]
I20240502 21:26:59.743276  2353 tablet_copy_service.cc:342] P 68f426685b4c4c90bda61ef52177cc22: Request end of tablet copy session 4e8c16d1a1484a41a54b5ddc198becd3-2779949906ff486da691638de7374050 received from {username='slave'} at 127.27.196.69:44867
I20240502 21:26:59.743458  2353 tablet_copy_service.cc:433] P 68f426685b4c4c90bda61ef52177cc22: ending tablet copy session 4e8c16d1a1484a41a54b5ddc198becd3-2779949906ff486da691638de7374050 on tablet 2779949906ff486da691638de7374050 with peer 4e8c16d1a1484a41a54b5ddc198becd3
I20240502 21:26:59.818061  4422 ts_tablet_manager.cc:902] T 6a38d2b4246e497fa91d2a0a8b127244 P 1193f1d0d2c843139d98458cf4035eb8: Initiating tablet copy from peer 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253)
I20240502 21:26:59.901196  4422 tablet_copy_client.cc:308] T 6a38d2b4246e497fa91d2a0a8b127244 P 1193f1d0d2c843139d98458cf4035eb8: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:36253
I20240502 21:26:59.903712  2353 tablet_copy_service.cc:140] P 68f426685b4c4c90bda61ef52177cc22: Received BeginTabletCopySession request for tablet 6a38d2b4246e497fa91d2a0a8b127244 from peer 1193f1d0d2c843139d98458cf4035eb8 ({username='slave'} at 127.27.196.70:56283)
I20240502 21:26:59.903905  2353 tablet_copy_service.cc:161] P 68f426685b4c4c90bda61ef52177cc22: Beginning new tablet copy session on tablet 6a38d2b4246e497fa91d2a0a8b127244 from peer 1193f1d0d2c843139d98458cf4035eb8 at {username='slave'} at 127.27.196.70:56283: session id = 1193f1d0d2c843139d98458cf4035eb8-6a38d2b4246e497fa91d2a0a8b127244
I20240502 21:26:59.911478  2353 tablet_copy_source_session.cc:205] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:59.915349  4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a38d2b4246e497fa91d2a0a8b127244. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.939524  4422 tablet_copy_client.cc:783] T 6a38d2b4246e497fa91d2a0a8b127244 P 1193f1d0d2c843139d98458cf4035eb8: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:59.939999  4422 tablet_copy_client.cc:647] T 6a38d2b4246e497fa91d2a0a8b127244 P 1193f1d0d2c843139d98458cf4035eb8: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:59.942855  4422 tablet_copy_client.cc:523] T 6a38d2b4246e497fa91d2a0a8b127244 P 1193f1d0d2c843139d98458cf4035eb8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:59.949129  4422 tablet_bootstrap.cc:492] T 6a38d2b4246e497fa91d2a0a8b127244 P 1193f1d0d2c843139d98458cf4035eb8: Bootstrap starting.
I20240502 21:27:00.007690  3552 consensus_queue.cc:237] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
I20240502 21:27:00.013482  3685 raft_consensus.cc:1203] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.81->[1.82-1.106]   Dedup: 1.88->[1.89-1.106]
I20240502 21:27:00.104359  2330 raft_consensus.cc:1261] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [term 2 FOLLOWER]: Refusing update from remote peer 13451be0f7134a2c89c0c9330177213e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 21:27:00.106413  4435 consensus_queue.cc:1035] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [LEADER]: Connected to new peer: Peer: permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 21:27:00.119781  3481 consensus_peers.cc:485] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e -> Peer d306b26127e04e8c85a98edf1b41978a (127.27.196.71:39583): Couldn't send request to peer d306b26127e04e8c85a98edf1b41978a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ae512d4d5874a909b1f4186c4b9785a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:00.237658  4398 tablet_bootstrap.cc:492] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e: Bootstrap replayed 1/1 log segments. Stats: ops{read=89 overwritten=0 applied=89 ignored=0} inserts{seen=774 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:00.238346  4398 tablet_bootstrap.cc:492] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e: Bootstrap complete.
I20240502 21:27:00.238790  4398 ts_tablet_manager.cc:1367] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e: Time spent bootstrapping tablet: real 0.503s	user 0.160s	sys 0.027s
I20240502 21:27:00.240562  4398 raft_consensus.cc:348] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } }
I20240502 21:27:00.240953  4398 raft_consensus.cc:729] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13451be0f7134a2c89c0c9330177213e, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.241329  4398 consensus_queue.cc:260] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 89, Last appended: 1.89, Last appended by leader: 89, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } }
I20240502 21:27:00.242142  4398 ts_tablet_manager.cc:1398] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 21:27:00.244009  2215 tablet_copy_service.cc:342] P d961985de7874e1990a19bb9b2da14c2: Request end of tablet copy session 13451be0f7134a2c89c0c9330177213e-6e1c5a5cc57243d7bf6e212e3a32dfe3 received from {username='slave'} at 127.27.196.68:52125
I20240502 21:27:00.244215  2215 tablet_copy_service.cc:433] P d961985de7874e1990a19bb9b2da14c2: ending tablet copy session 13451be0f7134a2c89c0c9330177213e-6e1c5a5cc57243d7bf6e212e3a32dfe3 on tablet 6e1c5a5cc57243d7bf6e212e3a32dfe3 with peer 13451be0f7134a2c89c0c9330177213e
I20240502 21:27:00.262638  4433 raft_consensus.cc:2932] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER d306b26127e04e8c85a98edf1b41978a (127.27.196.71) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:00.264106  2330 raft_consensus.cc:2932] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER d306b26127e04e8c85a98edf1b41978a (127.27.196.71) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:00.271234  2045 catalog_manager.cc:5484] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e reported cstate change: config changed from index -1 to 3, VOTER d306b26127e04e8c85a98edf1b41978a (127.27.196.71) added. New cstate: current_term: 2 leader_uuid: "13451be0f7134a2c89c0c9330177213e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:00.382546  4422 tablet_bootstrap.cc:492] T 6a38d2b4246e497fa91d2a0a8b127244 P 1193f1d0d2c843139d98458cf4035eb8: Bootstrap replayed 1/1 log segments. Stats: ops{read=71 overwritten=0 applied=70 ignored=0} inserts{seen=742 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 21:27:00.383666  4422 tablet_bootstrap.cc:492] T 6a38d2b4246e497fa91d2a0a8b127244 P 1193f1d0d2c843139d98458cf4035eb8: Bootstrap complete.
I20240502 21:27:00.384107  4422 ts_tablet_manager.cc:1367] T 6a38d2b4246e497fa91d2a0a8b127244 P 1193f1d0d2c843139d98458cf4035eb8: Time spent bootstrapping tablet: real 0.435s	user 0.139s	sys 0.021s
I20240502 21:27:00.385264  4422 raft_consensus.cc:348] T 6a38d2b4246e497fa91d2a0a8b127244 P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } }
I20240502 21:27:00.388685  4422 raft_consensus.cc:729] T 6a38d2b4246e497fa91d2a0a8b127244 P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1193f1d0d2c843139d98458cf4035eb8, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.389200  4422 consensus_queue.cc:260] T 6a38d2b4246e497fa91d2a0a8b127244 P 1193f1d0d2c843139d98458cf4035eb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 70, Last appended: 1.71, Last appended by leader: 71, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } }
I20240502 21:27:00.397615  4422 ts_tablet_manager.cc:1398] T 6a38d2b4246e497fa91d2a0a8b127244 P 1193f1d0d2c843139d98458cf4035eb8: Time spent starting tablet: real 0.013s	user 0.003s	sys 0.006s
I20240502 21:27:00.398871  2353 tablet_copy_service.cc:342] P 68f426685b4c4c90bda61ef52177cc22: Request end of tablet copy session 1193f1d0d2c843139d98458cf4035eb8-6a38d2b4246e497fa91d2a0a8b127244 received from {username='slave'} at 127.27.196.70:56283
I20240502 21:27:00.399070  2353 tablet_copy_service.cc:433] P 68f426685b4c4c90bda61ef52177cc22: ending tablet copy session 1193f1d0d2c843139d98458cf4035eb8-6a38d2b4246e497fa91d2a0a8b127244 on tablet 6a38d2b4246e497fa91d2a0a8b127244 with peer 1193f1d0d2c843139d98458cf4035eb8
I20240502 21:27:00.512708  3552 raft_consensus.cc:1203] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.80->[1.81-1.97]   Dedup: 1.89->[1.90-1.97]
I20240502 21:27:00.607090  4462 ts_tablet_manager.cc:902] T 5ae512d4d5874a909b1f4186c4b9785a P d306b26127e04e8c85a98edf1b41978a: Initiating tablet copy from peer 13451be0f7134a2c89c0c9330177213e (127.27.196.68:44575)
I20240502 21:27:00.626266  3817 raft_consensus.cc:1203] T 6a38d2b4246e497fa91d2a0a8b127244 P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.78]   Dedup: 1.71->[1.72-1.78]
I20240502 21:27:00.858489  4462 tablet_copy_client.cc:308] T 5ae512d4d5874a909b1f4186c4b9785a P d306b26127e04e8c85a98edf1b41978a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.68:44575
I20240502 21:27:00.900786  3573 tablet_copy_service.cc:140] P 13451be0f7134a2c89c0c9330177213e: Received BeginTabletCopySession request for tablet 5ae512d4d5874a909b1f4186c4b9785a from peer d306b26127e04e8c85a98edf1b41978a ({username='slave'} at 127.27.196.71:46819)
I20240502 21:27:00.901183  3573 tablet_copy_service.cc:161] P 13451be0f7134a2c89c0c9330177213e: Beginning new tablet copy session on tablet 5ae512d4d5874a909b1f4186c4b9785a from peer d306b26127e04e8c85a98edf1b41978a at {username='slave'} at 127.27.196.71:46819: session id = d306b26127e04e8c85a98edf1b41978a-5ae512d4d5874a909b1f4186c4b9785a
I20240502 21:27:00.911564  3573 tablet_copy_source_session.cc:205] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:00.913820  4462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae512d4d5874a909b1f4186c4b9785a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.921782  4462 tablet_copy_client.cc:783] T 5ae512d4d5874a909b1f4186c4b9785a P d306b26127e04e8c85a98edf1b41978a: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:00.923204  4462 tablet_copy_client.cc:647] T 5ae512d4d5874a909b1f4186c4b9785a P d306b26127e04e8c85a98edf1b41978a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:00.925698  4462 tablet_copy_client.cc:523] T 5ae512d4d5874a909b1f4186c4b9785a P d306b26127e04e8c85a98edf1b41978a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:00.931250  4462 tablet_bootstrap.cc:492] T 5ae512d4d5874a909b1f4186c4b9785a P d306b26127e04e8c85a98edf1b41978a: Bootstrap starting.
I20240502 21:27:00.948626  4462 tablet_bootstrap.cc:492] T 5ae512d4d5874a909b1f4186c4b9785a P d306b26127e04e8c85a98edf1b41978a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:00.956676  4462 tablet_bootstrap.cc:492] T 5ae512d4d5874a909b1f4186c4b9785a P d306b26127e04e8c85a98edf1b41978a: Bootstrap complete.
I20240502 21:27:00.957130  4462 ts_tablet_manager.cc:1367] T 5ae512d4d5874a909b1f4186c4b9785a P d306b26127e04e8c85a98edf1b41978a: Time spent bootstrapping tablet: real 0.026s	user 0.019s	sys 0.000s
I20240502 21:27:00.958474  4462 raft_consensus.cc:348] T 5ae512d4d5874a909b1f4186c4b9785a P d306b26127e04e8c85a98edf1b41978a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
I20240502 21:27:00.958828  4462 raft_consensus.cc:729] T 5ae512d4d5874a909b1f4186c4b9785a P d306b26127e04e8c85a98edf1b41978a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d306b26127e04e8c85a98edf1b41978a, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.959211  4462 consensus_queue.cc:260] T 5ae512d4d5874a909b1f4186c4b9785a P d306b26127e04e8c85a98edf1b41978a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
I20240502 21:27:00.979059  4462 ts_tablet_manager.cc:1398] T 5ae512d4d5874a909b1f4186c4b9785a P d306b26127e04e8c85a98edf1b41978a: Time spent starting tablet: real 0.022s	user 0.006s	sys 0.007s
I20240502 21:27:00.980429  3573 tablet_copy_service.cc:342] P 13451be0f7134a2c89c0c9330177213e: Request end of tablet copy session d306b26127e04e8c85a98edf1b41978a-5ae512d4d5874a909b1f4186c4b9785a received from {username='slave'} at 127.27.196.71:46819
I20240502 21:27:00.980670  3573 tablet_copy_service.cc:433] P 13451be0f7134a2c89c0c9330177213e: ending tablet copy session d306b26127e04e8c85a98edf1b41978a-5ae512d4d5874a909b1f4186c4b9785a on tablet 5ae512d4d5874a909b1f4186c4b9785a with peer d306b26127e04e8c85a98edf1b41978a
I20240502 21:27:01.034852  2329 consensus_queue.cc:237] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 1.89, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } }
I20240502 21:27:01.037379  2194 raft_consensus.cc:1261] T be2cbcd79c1e40f49701271bb247f4d0 P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Refusing update from remote peer 68f426685b4c4c90bda61ef52177cc22: Log matching property violated. Preceding OpId in replica: term: 1 index: 89. Preceding OpId from leader: term: 1 index: 90. (index mismatch)
I20240502 21:27:01.039356  3442 consensus_queue.cc:1035] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 90, Last known committed idx: 89, Time since last communication: 0.000s
W20240502 21:27:01.040798  2265 consensus_peers.cc:485] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22 -> Peer 4e8c16d1a1484a41a54b5ddc198becd3 (127.27.196.69:37783): Couldn't send request to peer 4e8c16d1a1484a41a54b5ddc198becd3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: be2cbcd79c1e40f49701271bb247f4d0. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:01.045225  3442 raft_consensus.cc:2932] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Committing config change with OpId 1.90: config changed from index -1 to 90, VOTER 4e8c16d1a1484a41a54b5ddc198becd3 (127.27.196.69) added. New config: { opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } }
I20240502 21:27:01.046633  2195 raft_consensus.cc:2932] T be2cbcd79c1e40f49701271bb247f4d0 P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Committing config change with OpId 1.90: config changed from index -1 to 90, VOTER 4e8c16d1a1484a41a54b5ddc198becd3 (127.27.196.69) added. New config: { opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } }
I20240502 21:27:01.055289  3966 raft_consensus.cc:1203] T 5ae512d4d5874a909b1f4186c4b9785a P d306b26127e04e8c85a98edf1b41978a [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3]   Dedup: 2.3->[]
I20240502 21:27:01.070159  2047 catalog_manager.cc:5484] T be2cbcd79c1e40f49701271bb247f4d0 P d961985de7874e1990a19bb9b2da14c2 reported cstate change: config changed from index -1 to 90, VOTER 4e8c16d1a1484a41a54b5ddc198becd3 (127.27.196.69) added. New cstate: current_term: 1 leader_uuid: "68f426685b4c4c90bda61ef52177cc22" committed_config { opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } }
I20240502 21:27:01.407748  2194 consensus_queue.cc:237] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 1.88, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } }
W20240502 21:27:01.411636  2130 consensus_peers.cc:485] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2 -> Peer 13451be0f7134a2c89c0c9330177213e (127.27.196.68:44575): Couldn't send request to peer 13451be0f7134a2c89c0c9330177213e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0264c436aafa422a9a54599e74e0221c. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:01.411818  2463 raft_consensus.cc:1261] T 0264c436aafa422a9a54599e74e0221c P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Refusing update from remote peer d961985de7874e1990a19bb9b2da14c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 88. Preceding OpId from leader: term: 1 index: 89. (index mismatch)
I20240502 21:27:01.412549  4428 consensus_queue.cc:1035] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 87, Time since last communication: 0.000s
I20240502 21:27:01.420143  4114 raft_consensus.cc:2932] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2 [term 1 LEADER]: Committing config change with OpId 1.89: config changed from index -1 to 89, VOTER 13451be0f7134a2c89c0c9330177213e (127.27.196.68) added. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } }
I20240502 21:27:01.429293  2466 raft_consensus.cc:2932] T 0264c436aafa422a9a54599e74e0221c P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Committing config change with OpId 1.89: config changed from index -1 to 89, VOTER 13451be0f7134a2c89c0c9330177213e (127.27.196.68) added. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } }
I20240502 21:27:01.429800  2050 catalog_manager.cc:5484] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2 reported cstate change: config changed from index -1 to 89, VOTER 13451be0f7134a2c89c0c9330177213e (127.27.196.68) added. New cstate: current_term: 1 leader_uuid: "d961985de7874e1990a19bb9b2da14c2" committed_config { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } }
I20240502 21:27:01.620986  4518 ts_tablet_manager.cc:902] T be2cbcd79c1e40f49701271bb247f4d0 P 4e8c16d1a1484a41a54b5ddc198becd3: Initiating tablet copy from peer 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253)
I20240502 21:27:01.760370  4518 tablet_copy_client.cc:308] T be2cbcd79c1e40f49701271bb247f4d0 P 4e8c16d1a1484a41a54b5ddc198becd3: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:36253
I20240502 21:27:01.761546  2353 tablet_copy_service.cc:140] P 68f426685b4c4c90bda61ef52177cc22: Received BeginTabletCopySession request for tablet be2cbcd79c1e40f49701271bb247f4d0 from peer 4e8c16d1a1484a41a54b5ddc198becd3 ({username='slave'} at 127.27.196.69:44867)
I20240502 21:27:01.761837  2353 tablet_copy_service.cc:161] P 68f426685b4c4c90bda61ef52177cc22: Beginning new tablet copy session on tablet be2cbcd79c1e40f49701271bb247f4d0 from peer 4e8c16d1a1484a41a54b5ddc198becd3 at {username='slave'} at 127.27.196.69:44867: session id = 4e8c16d1a1484a41a54b5ddc198becd3-be2cbcd79c1e40f49701271bb247f4d0
I20240502 21:27:01.768671  2353 tablet_copy_source_session.cc:205] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:01.820470  4518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be2cbcd79c1e40f49701271bb247f4d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.841024  4518 tablet_copy_client.cc:783] T be2cbcd79c1e40f49701271bb247f4d0 P 4e8c16d1a1484a41a54b5ddc198becd3: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:01.841413  4518 tablet_copy_client.cc:647] T be2cbcd79c1e40f49701271bb247f4d0 P 4e8c16d1a1484a41a54b5ddc198becd3: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:01.843858  4518 tablet_copy_client.cc:523] T be2cbcd79c1e40f49701271bb247f4d0 P 4e8c16d1a1484a41a54b5ddc198becd3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:01.847854  4518 tablet_bootstrap.cc:492] T be2cbcd79c1e40f49701271bb247f4d0 P 4e8c16d1a1484a41a54b5ddc198becd3: Bootstrap starting.
I20240502 21:27:01.947978  4530 ts_tablet_manager.cc:902] T 0264c436aafa422a9a54599e74e0221c P 13451be0f7134a2c89c0c9330177213e: Initiating tablet copy from peer d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025)
I20240502 21:27:01.990409  2192 consensus_queue.cc:237] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 138, Committed index: 138, Last appended: 1.139, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } }
I20240502 21:27:02.047055  4530 tablet_copy_client.cc:308] T 0264c436aafa422a9a54599e74e0221c P 13451be0f7134a2c89c0c9330177213e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:37025
I20240502 21:27:02.048452  2328 raft_consensus.cc:1261] T 068c9a3114f648d1a183c9b90f4d365e P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Refusing update from remote peer d961985de7874e1990a19bb9b2da14c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 139. Preceding OpId from leader: term: 1 index: 140. (index mismatch)
I20240502 21:27:02.053640  2215 tablet_copy_service.cc:140] P d961985de7874e1990a19bb9b2da14c2: Received BeginTabletCopySession request for tablet 0264c436aafa422a9a54599e74e0221c from peer 13451be0f7134a2c89c0c9330177213e ({username='slave'} at 127.27.196.68:52125)
I20240502 21:27:02.053897  2215 tablet_copy_service.cc:161] P d961985de7874e1990a19bb9b2da14c2: Beginning new tablet copy session on tablet 0264c436aafa422a9a54599e74e0221c from peer 13451be0f7134a2c89c0c9330177213e at {username='slave'} at 127.27.196.68:52125: session id = 13451be0f7134a2c89c0c9330177213e-0264c436aafa422a9a54599e74e0221c
I20240502 21:27:02.058063  2215 tablet_copy_source_session.cc:205] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:02.060483  4428 consensus_queue.cc:1035] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 140, Last known committed idx: 138, Time since last communication: 0.000s
I20240502 21:27:02.061268  4530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0264c436aafa422a9a54599e74e0221c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.071162  4530 tablet_copy_client.cc:783] T 0264c436aafa422a9a54599e74e0221c P 13451be0f7134a2c89c0c9330177213e: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:02.071524  4530 tablet_copy_client.cc:647] T 0264c436aafa422a9a54599e74e0221c P 13451be0f7134a2c89c0c9330177213e: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:02.086608  4530 tablet_copy_client.cc:523] T 0264c436aafa422a9a54599e74e0221c P 13451be0f7134a2c89c0c9330177213e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:02.091490  4530 tablet_bootstrap.cc:492] T 0264c436aafa422a9a54599e74e0221c P 13451be0f7134a2c89c0c9330177213e: Bootstrap starting.
W20240502 21:27:02.094228  2130 consensus_peers.cc:485] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 -> Peer 1193f1d0d2c843139d98458cf4035eb8 (127.27.196.70:46179): Couldn't send request to peer 1193f1d0d2c843139d98458cf4035eb8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 068c9a3114f648d1a183c9b90f4d365e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:02.107160  4099 raft_consensus.cc:2932] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [term 1 LEADER]: Committing config change with OpId 1.140: config changed from index -1 to 140, VOTER 1193f1d0d2c843139d98458cf4035eb8 (127.27.196.70) added. New config: { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } }
I20240502 21:27:02.111963  2328 raft_consensus.cc:2932] T 068c9a3114f648d1a183c9b90f4d365e P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Committing config change with OpId 1.140: config changed from index -1 to 140, VOTER 1193f1d0d2c843139d98458cf4035eb8 (127.27.196.70) added. New config: { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } }
I20240502 21:27:02.190696  2050 catalog_manager.cc:5484] T 068c9a3114f648d1a183c9b90f4d365e P 68f426685b4c4c90bda61ef52177cc22 reported cstate change: config changed from index -1 to 140, VOTER 1193f1d0d2c843139d98458cf4035eb8 (127.27.196.70) added. New cstate: current_term: 1 leader_uuid: "d961985de7874e1990a19bb9b2da14c2" committed_config { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } }
I20240502 21:27:02.388896  4518 tablet_bootstrap.cc:492] T be2cbcd79c1e40f49701271bb247f4d0 P 4e8c16d1a1484a41a54b5ddc198becd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=98 overwritten=0 applied=98 ignored=0} inserts{seen=770 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:02.395010  4518 tablet_bootstrap.cc:492] T be2cbcd79c1e40f49701271bb247f4d0 P 4e8c16d1a1484a41a54b5ddc198becd3: Bootstrap complete.
I20240502 21:27:02.395453  4518 ts_tablet_manager.cc:1367] T be2cbcd79c1e40f49701271bb247f4d0 P 4e8c16d1a1484a41a54b5ddc198becd3: Time spent bootstrapping tablet: real 0.548s	user 0.176s	sys 0.032s
I20240502 21:27:02.396623  4518 raft_consensus.cc:348] T be2cbcd79c1e40f49701271bb247f4d0 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } }
I20240502 21:27:02.396929  4518 raft_consensus.cc:729] T be2cbcd79c1e40f49701271bb247f4d0 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e8c16d1a1484a41a54b5ddc198becd3, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.404877  4518 consensus_queue.cc:260] T be2cbcd79c1e40f49701271bb247f4d0 P 4e8c16d1a1484a41a54b5ddc198becd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 98, Last appended: 1.98, Last appended by leader: 98, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } }
I20240502 21:27:02.432160  4518 ts_tablet_manager.cc:1398] T be2cbcd79c1e40f49701271bb247f4d0 P 4e8c16d1a1484a41a54b5ddc198becd3: Time spent starting tablet: real 0.036s	user 0.008s	sys 0.001s
I20240502 21:27:02.433607  2353 tablet_copy_service.cc:342] P 68f426685b4c4c90bda61ef52177cc22: Request end of tablet copy session 4e8c16d1a1484a41a54b5ddc198becd3-be2cbcd79c1e40f49701271bb247f4d0 received from {username='slave'} at 127.27.196.69:44867
I20240502 21:27:02.433791  2353 tablet_copy_service.cc:433] P 68f426685b4c4c90bda61ef52177cc22: ending tablet copy session 4e8c16d1a1484a41a54b5ddc198becd3-be2cbcd79c1e40f49701271bb247f4d0 on tablet be2cbcd79c1e40f49701271bb247f4d0 with peer 4e8c16d1a1484a41a54b5ddc198becd3
I20240502 21:27:02.475960  4530 tablet_bootstrap.cc:492] T 0264c436aafa422a9a54599e74e0221c P 13451be0f7134a2c89c0c9330177213e: Bootstrap replayed 1/1 log segments. Stats: ops{read=100 overwritten=0 applied=100 ignored=0} inserts{seen=731 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:02.476686  4530 tablet_bootstrap.cc:492] T 0264c436aafa422a9a54599e74e0221c P 13451be0f7134a2c89c0c9330177213e: Bootstrap complete.
I20240502 21:27:02.477061  4530 ts_tablet_manager.cc:1367] T 0264c436aafa422a9a54599e74e0221c P 13451be0f7134a2c89c0c9330177213e: Time spent bootstrapping tablet: real 0.386s	user 0.172s	sys 0.016s
I20240502 21:27:02.478008  4530 raft_consensus.cc:348] T 0264c436aafa422a9a54599e74e0221c P 13451be0f7134a2c89c0c9330177213e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } }
I20240502 21:27:02.478480  4530 raft_consensus.cc:729] T 0264c436aafa422a9a54599e74e0221c P 13451be0f7134a2c89c0c9330177213e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13451be0f7134a2c89c0c9330177213e, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.478880  4530 consensus_queue.cc:260] T 0264c436aafa422a9a54599e74e0221c P 13451be0f7134a2c89c0c9330177213e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 100, Last appended: 1.100, Last appended by leader: 100, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } }
I20240502 21:27:02.479667  4530 ts_tablet_manager.cc:1398] T 0264c436aafa422a9a54599e74e0221c P 13451be0f7134a2c89c0c9330177213e: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 21:27:02.483942  2215 tablet_copy_service.cc:342] P d961985de7874e1990a19bb9b2da14c2: Request end of tablet copy session 13451be0f7134a2c89c0c9330177213e-0264c436aafa422a9a54599e74e0221c received from {username='slave'} at 127.27.196.68:52125
I20240502 21:27:02.484158  2215 tablet_copy_service.cc:433] P d961985de7874e1990a19bb9b2da14c2: ending tablet copy session 13451be0f7134a2c89c0c9330177213e-0264c436aafa422a9a54599e74e0221c on tablet 0264c436aafa422a9a54599e74e0221c with peer 13451be0f7134a2c89c0c9330177213e
I20240502 21:27:02.490906  4554 ts_tablet_manager.cc:902] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8: Initiating tablet copy from peer d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025)
I20240502 21:27:02.574735  3685 raft_consensus.cc:1203] T be2cbcd79c1e40f49701271bb247f4d0 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.89->[1.90-1.104]   Dedup: 1.98->[1.99-1.104]
I20240502 21:27:02.774062  4554 tablet_copy_client.cc:308] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:37025
I20240502 21:27:02.780772  3551 raft_consensus.cc:1203] T 0264c436aafa422a9a54599e74e0221c P 13451be0f7134a2c89c0c9330177213e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.88->[1.89-1.109]   Dedup: 1.100->[1.101-1.109]
I20240502 21:27:02.931835  2215 tablet_copy_service.cc:140] P d961985de7874e1990a19bb9b2da14c2: Received BeginTabletCopySession request for tablet 068c9a3114f648d1a183c9b90f4d365e from peer 1193f1d0d2c843139d98458cf4035eb8 ({username='slave'} at 127.27.196.70:33233)
I20240502 21:27:02.932058  2215 tablet_copy_service.cc:161] P d961985de7874e1990a19bb9b2da14c2: Beginning new tablet copy session on tablet 068c9a3114f648d1a183c9b90f4d365e from peer 1193f1d0d2c843139d98458cf4035eb8 at {username='slave'} at 127.27.196.70:33233: session id = 1193f1d0d2c843139d98458cf4035eb8-068c9a3114f648d1a183c9b90f4d365e
I20240502 21:27:02.936281  2215 tablet_copy_source_session.cc:205] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:02.943223  4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 068c9a3114f648d1a183c9b90f4d365e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.951081  4554 tablet_copy_client.cc:783] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:02.951519  4554 tablet_copy_client.cc:647] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:02.957049  4554 tablet_copy_client.cc:523] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:02.998059  4554 tablet_bootstrap.cc:492] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8: Bootstrap starting.
I20240502 21:27:03.011957  2466 consensus_queue.cc:237] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 1.123, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
W20240502 21:27:03.017654  2398 consensus_peers.cc:485] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 -> Peer d306b26127e04e8c85a98edf1b41978a (127.27.196.71:39583): Couldn't send request to peer d306b26127e04e8c85a98edf1b41978a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0306f5bfea0a49eca06554a40ba50d34. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:03.018237  2192 raft_consensus.cc:1261] T 0306f5bfea0a49eca06554a40ba50d34 P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Refusing update from remote peer c616208720db4b14a9219cae8f2b7d49: Log matching property violated. Preceding OpId in replica: term: 1 index: 123. Preceding OpId from leader: term: 1 index: 124. (index mismatch)
I20240502 21:27:03.019309  4560 consensus_queue.cc:1035] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20240502 21:27:03.033510  4560 raft_consensus.cc:2932] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [term 1 LEADER]: Committing config change with OpId 1.124: config changed from index -1 to 124, VOTER d306b26127e04e8c85a98edf1b41978a (127.27.196.71) added. New config: { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:03.037269  2192 raft_consensus.cc:2932] T 0306f5bfea0a49eca06554a40ba50d34 P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Committing config change with OpId 1.124: config changed from index -1 to 124, VOTER d306b26127e04e8c85a98edf1b41978a (127.27.196.71) added. New config: { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:03.046707  2050 catalog_manager.cc:5484] T 0306f5bfea0a49eca06554a40ba50d34 P d961985de7874e1990a19bb9b2da14c2 reported cstate change: config changed from index -1 to 124, VOTER d306b26127e04e8c85a98edf1b41978a (127.27.196.71) added. New cstate: current_term: 1 leader_uuid: "c616208720db4b14a9219cae8f2b7d49" committed_config { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:03.562409  2462 consensus_queue.cc:237] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 1.103, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } }
I20240502 21:27:03.601509  2195 raft_consensus.cc:1261] T d4392d14cf254b72bf506f9c6cfdd55d P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Refusing update from remote peer c616208720db4b14a9219cae8f2b7d49: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
I20240502 21:27:03.607437  4560 consensus_queue.cc:1035] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 102, Time since last communication: 0.000s
I20240502 21:27:03.612293  4606 ts_tablet_manager.cc:902] T 0306f5bfea0a49eca06554a40ba50d34 P d306b26127e04e8c85a98edf1b41978a: Initiating tablet copy from peer c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687)
I20240502 21:27:03.680519  4560 raft_consensus.cc:2932] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [term 1 LEADER]: Committing config change with OpId 1.104: config changed from index -1 to 104, VOTER 4e8c16d1a1484a41a54b5ddc198becd3 (127.27.196.69) added. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } }
I20240502 21:27:03.681612  2195 raft_consensus.cc:2932] T d4392d14cf254b72bf506f9c6cfdd55d P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index -1 to 104, VOTER 4e8c16d1a1484a41a54b5ddc198becd3 (127.27.196.69) added. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } }
I20240502 21:27:03.689060  4554 tablet_bootstrap.cc:492] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8: Bootstrap replayed 1/1 log segments. Stats: ops{read=151 overwritten=0 applied=151 ignored=0} inserts{seen=823 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:03.689589  4554 tablet_bootstrap.cc:492] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8: Bootstrap complete.
I20240502 21:27:03.689934  4554 ts_tablet_manager.cc:1367] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8: Time spent bootstrapping tablet: real 0.692s	user 0.322s	sys 0.134s
I20240502 21:27:03.689886  2046 catalog_manager.cc:5484] T d4392d14cf254b72bf506f9c6cfdd55d P d961985de7874e1990a19bb9b2da14c2 reported cstate change: config changed from index -1 to 104, VOTER 4e8c16d1a1484a41a54b5ddc198becd3 (127.27.196.69) added. New cstate: current_term: 1 leader_uuid: "c616208720db4b14a9219cae8f2b7d49" committed_config { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } }
I20240502 21:27:03.690758  4554 raft_consensus.cc:348] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } }
I20240502 21:27:03.691001  4554 raft_consensus.cc:729] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1193f1d0d2c843139d98458cf4035eb8, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.691262  4554 consensus_queue.cc:260] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 151, Last appended: 1.151, Last appended by leader: 151, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } }
W20240502 21:27:03.733811  2401 consensus_peers.cc:485] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 -> Peer 4e8c16d1a1484a41a54b5ddc198becd3 (127.27.196.69:37783): Couldn't send request to peer 4e8c16d1a1484a41a54b5ddc198becd3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4392d14cf254b72bf506f9c6cfdd55d. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:03.747074  4554 ts_tablet_manager.cc:1398] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8: Time spent starting tablet: real 0.057s	user 0.024s	sys 0.023s
I20240502 21:27:03.765224  2215 tablet_copy_service.cc:342] P d961985de7874e1990a19bb9b2da14c2: Request end of tablet copy session 1193f1d0d2c843139d98458cf4035eb8-068c9a3114f648d1a183c9b90f4d365e received from {username='slave'} at 127.27.196.70:33233
I20240502 21:27:03.765430  2215 tablet_copy_service.cc:433] P d961985de7874e1990a19bb9b2da14c2: ending tablet copy session 1193f1d0d2c843139d98458cf4035eb8-068c9a3114f648d1a183c9b90f4d365e on tablet 068c9a3114f648d1a183c9b90f4d365e with peer 1193f1d0d2c843139d98458cf4035eb8
I20240502 21:27:03.833369  4606 tablet_copy_client.cc:308] T 0306f5bfea0a49eca06554a40ba50d34 P d306b26127e04e8c85a98edf1b41978a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:46687
I20240502 21:27:03.836282  2487 tablet_copy_service.cc:140] P c616208720db4b14a9219cae8f2b7d49: Received BeginTabletCopySession request for tablet 0306f5bfea0a49eca06554a40ba50d34 from peer d306b26127e04e8c85a98edf1b41978a ({username='slave'} at 127.27.196.71:42883)
I20240502 21:27:03.836525  2487 tablet_copy_service.cc:161] P c616208720db4b14a9219cae8f2b7d49: Beginning new tablet copy session on tablet 0306f5bfea0a49eca06554a40ba50d34 from peer d306b26127e04e8c85a98edf1b41978a at {username='slave'} at 127.27.196.71:42883: session id = d306b26127e04e8c85a98edf1b41978a-0306f5bfea0a49eca06554a40ba50d34
I20240502 21:27:03.840485  2487 tablet_copy_source_session.cc:205] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:03.842346  4606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0306f5bfea0a49eca06554a40ba50d34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.861647  4606 tablet_copy_client.cc:783] T 0306f5bfea0a49eca06554a40ba50d34 P d306b26127e04e8c85a98edf1b41978a: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:03.862254  4606 tablet_copy_client.cc:647] T 0306f5bfea0a49eca06554a40ba50d34 P d306b26127e04e8c85a98edf1b41978a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:03.866175  4606 tablet_copy_client.cc:523] T 0306f5bfea0a49eca06554a40ba50d34 P d306b26127e04e8c85a98edf1b41978a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:03.872857  4606 tablet_bootstrap.cc:492] T 0306f5bfea0a49eca06554a40ba50d34 P d306b26127e04e8c85a98edf1b41978a: Bootstrap starting.
I20240502 21:27:03.974226  2195 consensus_queue.cc:237] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 1.121, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
I20240502 21:27:03.983460  2463 raft_consensus.cc:1261] T e93810d704674fc4ad4d96501fb5ea42 P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Refusing update from remote peer d961985de7874e1990a19bb9b2da14c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 121. Preceding OpId from leader: term: 1 index: 122. (index mismatch)
I20240502 21:27:03.985157  4539 consensus_queue.cc:1035] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20240502 21:27:03.998381  3817 raft_consensus.cc:1203] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.139->[1.140-1.161]   Dedup: 1.151->[1.152-1.161]
W20240502 21:27:04.013828  2127 consensus_peers.cc:485] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 -> Peer d306b26127e04e8c85a98edf1b41978a (127.27.196.71:39583): Couldn't send request to peer d306b26127e04e8c85a98edf1b41978a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e93810d704674fc4ad4d96501fb5ea42. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:04.013943  4114 raft_consensus.cc:2932] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [term 1 LEADER]: Committing config change with OpId 1.122: config changed from index -1 to 122, VOTER d306b26127e04e8c85a98edf1b41978a (127.27.196.71) added. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:04.019405  2462 raft_consensus.cc:2932] T e93810d704674fc4ad4d96501fb5ea42 P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Committing config change with OpId 1.122: config changed from index -1 to 122, VOTER d306b26127e04e8c85a98edf1b41978a (127.27.196.71) added. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:04.080689  2046 catalog_manager.cc:5484] T e93810d704674fc4ad4d96501fb5ea42 P c616208720db4b14a9219cae8f2b7d49 reported cstate change: config changed from index -1 to 122, VOTER d306b26127e04e8c85a98edf1b41978a (127.27.196.71) added. New cstate: current_term: 1 leader_uuid: "d961985de7874e1990a19bb9b2da14c2" committed_config { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:04.224478  4606 tablet_bootstrap.cc:492] T 0306f5bfea0a49eca06554a40ba50d34 P d306b26127e04e8c85a98edf1b41978a: Bootstrap replayed 1/1 log segments. Stats: ops{read=132 overwritten=0 applied=132 ignored=0} inserts{seen=817 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:04.225143  4606 tablet_bootstrap.cc:492] T 0306f5bfea0a49eca06554a40ba50d34 P d306b26127e04e8c85a98edf1b41978a: Bootstrap complete.
I20240502 21:27:04.225526  4606 ts_tablet_manager.cc:1367] T 0306f5bfea0a49eca06554a40ba50d34 P d306b26127e04e8c85a98edf1b41978a: Time spent bootstrapping tablet: real 0.353s	user 0.228s	sys 0.019s
I20240502 21:27:04.226601  4606 raft_consensus.cc:348] T 0306f5bfea0a49eca06554a40ba50d34 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
I20240502 21:27:04.226992  4606 raft_consensus.cc:729] T 0306f5bfea0a49eca06554a40ba50d34 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d306b26127e04e8c85a98edf1b41978a, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.227329  4606 consensus_queue.cc:260] T 0306f5bfea0a49eca06554a40ba50d34 P d306b26127e04e8c85a98edf1b41978a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 132, Last appended: 1.132, Last appended by leader: 132, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
I20240502 21:27:04.234947  4606 ts_tablet_manager.cc:1398] T 0306f5bfea0a49eca06554a40ba50d34 P d306b26127e04e8c85a98edf1b41978a: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20240502 21:27:04.237685  2487 tablet_copy_service.cc:342] P c616208720db4b14a9219cae8f2b7d49: Request end of tablet copy session d306b26127e04e8c85a98edf1b41978a-0306f5bfea0a49eca06554a40ba50d34 received from {username='slave'} at 127.27.196.71:42883
I20240502 21:27:04.237866  2487 tablet_copy_service.cc:433] P c616208720db4b14a9219cae8f2b7d49: ending tablet copy session d306b26127e04e8c85a98edf1b41978a-0306f5bfea0a49eca06554a40ba50d34 on tablet 0306f5bfea0a49eca06554a40ba50d34 with peer d306b26127e04e8c85a98edf1b41978a
I20240502 21:27:04.241636  4637 ts_tablet_manager.cc:902] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3: Initiating tablet copy from peer c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687)
I20240502 21:27:04.274953  4637 tablet_copy_client.cc:308] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:46687
I20240502 21:27:04.342242  2487 tablet_copy_service.cc:140] P c616208720db4b14a9219cae8f2b7d49: Received BeginTabletCopySession request for tablet d4392d14cf254b72bf506f9c6cfdd55d from peer 4e8c16d1a1484a41a54b5ddc198becd3 ({username='slave'} at 127.27.196.69:41937)
I20240502 21:27:04.342464  2487 tablet_copy_service.cc:161] P c616208720db4b14a9219cae8f2b7d49: Beginning new tablet copy session on tablet d4392d14cf254b72bf506f9c6cfdd55d from peer 4e8c16d1a1484a41a54b5ddc198becd3 at {username='slave'} at 127.27.196.69:41937: session id = 4e8c16d1a1484a41a54b5ddc198becd3-d4392d14cf254b72bf506f9c6cfdd55d
I20240502 21:27:04.344841  2487 tablet_copy_source_session.cc:205] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:04.357887  4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4392d14cf254b72bf506f9c6cfdd55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.365192  4637 tablet_copy_client.cc:783] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:04.365623  4637 tablet_copy_client.cc:647] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:04.391386  4637 tablet_copy_client.cc:523] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:04.397132  4637 tablet_bootstrap.cc:492] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3: Bootstrap starting.
I20240502 21:27:04.425951  3966 raft_consensus.cc:1203] T 0306f5bfea0a49eca06554a40ba50d34 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.123->[1.124-1.140]   Dedup: 1.132->[1.133-1.140]
I20240502 21:27:04.549783  4606 ts_tablet_manager.cc:902] T e93810d704674fc4ad4d96501fb5ea42 P d306b26127e04e8c85a98edf1b41978a: Initiating tablet copy from peer d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025)
I20240502 21:27:04.669384  4606 tablet_copy_client.cc:308] T e93810d704674fc4ad4d96501fb5ea42 P d306b26127e04e8c85a98edf1b41978a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:37025
I20240502 21:27:04.982659  2215 tablet_copy_service.cc:140] P d961985de7874e1990a19bb9b2da14c2: Received BeginTabletCopySession request for tablet e93810d704674fc4ad4d96501fb5ea42 from peer d306b26127e04e8c85a98edf1b41978a ({username='slave'} at 127.27.196.71:55737)
I20240502 21:27:04.982906  2215 tablet_copy_service.cc:161] P d961985de7874e1990a19bb9b2da14c2: Beginning new tablet copy session on tablet e93810d704674fc4ad4d96501fb5ea42 from peer d306b26127e04e8c85a98edf1b41978a at {username='slave'} at 127.27.196.71:55737: session id = d306b26127e04e8c85a98edf1b41978a-e93810d704674fc4ad4d96501fb5ea42
I20240502 21:27:04.988559  2215 tablet_copy_source_session.cc:205] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:04.993322  4606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93810d704674fc4ad4d96501fb5ea42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.003228  4606 tablet_copy_client.cc:783] T e93810d704674fc4ad4d96501fb5ea42 P d306b26127e04e8c85a98edf1b41978a: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:05.008294  4606 tablet_copy_client.cc:647] T e93810d704674fc4ad4d96501fb5ea42 P d306b26127e04e8c85a98edf1b41978a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:05.014025  4606 tablet_copy_client.cc:523] T e93810d704674fc4ad4d96501fb5ea42 P d306b26127e04e8c85a98edf1b41978a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:05.019274  4606 tablet_bootstrap.cc:492] T e93810d704674fc4ad4d96501fb5ea42 P d306b26127e04e8c85a98edf1b41978a: Bootstrap starting.
I20240502 21:27:05.172695  2328 consensus_queue.cc:237] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 154, Committed index: 154, Last appended: 1.154, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 155 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
I20240502 21:27:05.181404  2466 raft_consensus.cc:1261] T dc17a741596a4cb59692b7f13cf254e7 P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Refusing update from remote peer 68f426685b4c4c90bda61ef52177cc22: Log matching property violated. Preceding OpId in replica: term: 1 index: 154. Preceding OpId from leader: term: 1 index: 155. (index mismatch)
I20240502 21:27:05.183147  4485 consensus_queue.cc:1035] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 155, Last known committed idx: 154, Time since last communication: 0.000s
I20240502 21:27:05.190235  4562 raft_consensus.cc:2932] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Committing config change with OpId 1.155: config changed from index -1 to 155, VOTER d306b26127e04e8c85a98edf1b41978a (127.27.196.71) added. New config: { opid_index: 155 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:05.198154  2466 raft_consensus.cc:2932] T dc17a741596a4cb59692b7f13cf254e7 P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Committing config change with OpId 1.155: config changed from index -1 to 155, VOTER d306b26127e04e8c85a98edf1b41978a (127.27.196.71) added. New config: { opid_index: 155 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:05.204332  2050 catalog_manager.cc:5484] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22 reported cstate change: config changed from index -1 to 155, VOTER d306b26127e04e8c85a98edf1b41978a (127.27.196.71) added. New cstate: current_term: 1 leader_uuid: "68f426685b4c4c90bda61ef52177cc22" committed_config { opid_index: 155 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
W20240502 21:27:05.294032  2262 consensus_peers.cc:485] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22 -> Peer d306b26127e04e8c85a98edf1b41978a (127.27.196.71:39583): Couldn't send request to peer d306b26127e04e8c85a98edf1b41978a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc17a741596a4cb59692b7f13cf254e7. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:05.444610  4606 tablet_bootstrap.cc:492] T e93810d704674fc4ad4d96501fb5ea42 P d306b26127e04e8c85a98edf1b41978a: Bootstrap replayed 1/1 log segments. Stats: ops{read=132 overwritten=0 applied=131 ignored=0} inserts{seen=762 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 21:27:05.445346  4606 tablet_bootstrap.cc:492] T e93810d704674fc4ad4d96501fb5ea42 P d306b26127e04e8c85a98edf1b41978a: Bootstrap complete.
I20240502 21:27:05.445765  4606 ts_tablet_manager.cc:1367] T e93810d704674fc4ad4d96501fb5ea42 P d306b26127e04e8c85a98edf1b41978a: Time spent bootstrapping tablet: real 0.427s	user 0.239s	sys 0.012s
I20240502 21:27:05.446832  4606 raft_consensus.cc:348] T e93810d704674fc4ad4d96501fb5ea42 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
I20240502 21:27:05.447587  4606 raft_consensus.cc:729] T e93810d704674fc4ad4d96501fb5ea42 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d306b26127e04e8c85a98edf1b41978a, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.448005  4606 consensus_queue.cc:260] T e93810d704674fc4ad4d96501fb5ea42 P d306b26127e04e8c85a98edf1b41978a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 131, Last appended: 1.132, Last appended by leader: 132, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
I20240502 21:27:05.452658  4637 tablet_bootstrap.cc:492] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=109 overwritten=0 applied=109 ignored=0} inserts{seen=794 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:05.466584  4637 tablet_bootstrap.cc:492] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3: Bootstrap complete.
I20240502 21:27:05.467088  4637 ts_tablet_manager.cc:1367] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3: Time spent bootstrapping tablet: real 1.070s	user 0.186s	sys 0.040s
I20240502 21:27:05.468183  4637 raft_consensus.cc:348] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } }
I20240502 21:27:05.470731  4637 raft_consensus.cc:729] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e8c16d1a1484a41a54b5ddc198becd3, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.473407  4637 consensus_queue.cc:260] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 109, Last appended: 1.109, Last appended by leader: 109, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } }
I20240502 21:27:05.555830  4606 ts_tablet_manager.cc:1398] T e93810d704674fc4ad4d96501fb5ea42 P d306b26127e04e8c85a98edf1b41978a: Time spent starting tablet: real 0.110s	user 0.048s	sys 0.045s
I20240502 21:27:05.562636  4637 ts_tablet_manager.cc:1398] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3: Time spent starting tablet: real 0.095s	user 0.003s	sys 0.002s
I20240502 21:27:05.563731  2215 tablet_copy_service.cc:342] P d961985de7874e1990a19bb9b2da14c2: Request end of tablet copy session d306b26127e04e8c85a98edf1b41978a-e93810d704674fc4ad4d96501fb5ea42 received from {username='slave'} at 127.27.196.71:55737
I20240502 21:27:05.563916  2215 tablet_copy_service.cc:433] P d961985de7874e1990a19bb9b2da14c2: ending tablet copy session d306b26127e04e8c85a98edf1b41978a-e93810d704674fc4ad4d96501fb5ea42 on tablet e93810d704674fc4ad4d96501fb5ea42 with peer d306b26127e04e8c85a98edf1b41978a
I20240502 21:27:05.569718  2487 tablet_copy_service.cc:342] P c616208720db4b14a9219cae8f2b7d49: Request end of tablet copy session 4e8c16d1a1484a41a54b5ddc198becd3-d4392d14cf254b72bf506f9c6cfdd55d received from {username='slave'} at 127.27.196.69:41937
I20240502 21:27:05.569890  2487 tablet_copy_service.cc:433] P c616208720db4b14a9219cae8f2b7d49: ending tablet copy session 4e8c16d1a1484a41a54b5ddc198becd3-d4392d14cf254b72bf506f9c6cfdd55d on tablet d4392d14cf254b72bf506f9c6cfdd55d with peer 4e8c16d1a1484a41a54b5ddc198becd3
I20240502 21:27:05.653131  3966 raft_consensus.cc:1203] T e93810d704674fc4ad4d96501fb5ea42 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.121->[1.122-1.140]   Dedup: 1.132->[1.133-1.140]
I20240502 21:27:05.731686  4606 ts_tablet_manager.cc:902] T dc17a741596a4cb59692b7f13cf254e7 P d306b26127e04e8c85a98edf1b41978a: Initiating tablet copy from peer 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253)
I20240502 21:27:05.801177  4606 tablet_copy_client.cc:308] T dc17a741596a4cb59692b7f13cf254e7 P d306b26127e04e8c85a98edf1b41978a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:36253
I20240502 21:27:05.897050  2353 tablet_copy_service.cc:140] P 68f426685b4c4c90bda61ef52177cc22: Received BeginTabletCopySession request for tablet dc17a741596a4cb59692b7f13cf254e7 from peer d306b26127e04e8c85a98edf1b41978a ({username='slave'} at 127.27.196.71:34799)
I20240502 21:27:05.897243  2353 tablet_copy_service.cc:161] P 68f426685b4c4c90bda61ef52177cc22: Beginning new tablet copy session on tablet dc17a741596a4cb59692b7f13cf254e7 from peer d306b26127e04e8c85a98edf1b41978a at {username='slave'} at 127.27.196.71:34799: session id = d306b26127e04e8c85a98edf1b41978a-dc17a741596a4cb59692b7f13cf254e7
I20240502 21:27:05.912786  2353 tablet_copy_source_session.cc:205] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:05.931525  4606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc17a741596a4cb59692b7f13cf254e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.941601  4606 tablet_copy_client.cc:783] T dc17a741596a4cb59692b7f13cf254e7 P d306b26127e04e8c85a98edf1b41978a: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:05.942014  4606 tablet_copy_client.cc:647] T dc17a741596a4cb59692b7f13cf254e7 P d306b26127e04e8c85a98edf1b41978a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:05.946234  4606 tablet_copy_client.cc:523] T dc17a741596a4cb59692b7f13cf254e7 P d306b26127e04e8c85a98edf1b41978a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:05.958182  4606 tablet_bootstrap.cc:492] T dc17a741596a4cb59692b7f13cf254e7 P d306b26127e04e8c85a98edf1b41978a: Bootstrap starting.
I20240502 21:27:06.000696  3683 raft_consensus.cc:1203] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.103->[1.104-1.125]   Dedup: 1.109->[1.110-1.125]
W20240502 21:27:06.146958  2400 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.65:37025, user_credentials={real_user=slave}} blocked reactor thread for 59634.8us
I20240502 21:27:06.470053  4606 tablet_bootstrap.cc:492] T dc17a741596a4cb59692b7f13cf254e7 P d306b26127e04e8c85a98edf1b41978a: Bootstrap replayed 1/1 log segments. Stats: ops{read=168 overwritten=0 applied=168 ignored=0} inserts{seen=799 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:06.470671  4606 tablet_bootstrap.cc:492] T dc17a741596a4cb59692b7f13cf254e7 P d306b26127e04e8c85a98edf1b41978a: Bootstrap complete.
I20240502 21:27:06.471102  4606 ts_tablet_manager.cc:1367] T dc17a741596a4cb59692b7f13cf254e7 P d306b26127e04e8c85a98edf1b41978a: Time spent bootstrapping tablet: real 0.513s	user 0.277s	sys 0.087s
I20240502 21:27:06.472079  4606 raft_consensus.cc:348] T dc17a741596a4cb59692b7f13cf254e7 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 155 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
I20240502 21:27:06.472436  4606 raft_consensus.cc:729] T dc17a741596a4cb59692b7f13cf254e7 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d306b26127e04e8c85a98edf1b41978a, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.472769  4606 consensus_queue.cc:260] T dc17a741596a4cb59692b7f13cf254e7 P d306b26127e04e8c85a98edf1b41978a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 168, Last appended: 1.168, Last appended by leader: 168, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 155 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
I20240502 21:27:06.496827  4606 ts_tablet_manager.cc:1398] T dc17a741596a4cb59692b7f13cf254e7 P d306b26127e04e8c85a98edf1b41978a: Time spent starting tablet: real 0.025s	user 0.007s	sys 0.016s
I20240502 21:27:06.500671  2353 tablet_copy_service.cc:342] P 68f426685b4c4c90bda61ef52177cc22: Request end of tablet copy session d306b26127e04e8c85a98edf1b41978a-dc17a741596a4cb59692b7f13cf254e7 received from {username='slave'} at 127.27.196.71:34799
I20240502 21:27:06.500947  2353 tablet_copy_service.cc:433] P 68f426685b4c4c90bda61ef52177cc22: ending tablet copy session d306b26127e04e8c85a98edf1b41978a-dc17a741596a4cb59692b7f13cf254e7 on tablet dc17a741596a4cb59692b7f13cf254e7 with peer d306b26127e04e8c85a98edf1b41978a
I20240502 21:27:06.689750  3963 raft_consensus.cc:1203] T dc17a741596a4cb59692b7f13cf254e7 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.154->[1.155-1.178]   Dedup: 1.168->[1.169-1.178]
I20240502 21:27:06.758975  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:06.760231  3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:27:06.763715  2263 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.67:46687, user_credentials={real_user=slave}} blocked reactor thread for 44662.1us
I20240502 21:27:06.848613  3665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:06.924289  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:07.004689  2308 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:27:07.012008  2174 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:07.178022  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 21:27:07.493006  2130 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.68:44575, user_credentials={real_user=slave}} blocked reactor thread for 235039us
I20240502 21:27:08.258767  2329 consensus_queue.cc:237] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 196, Committed index: 196, Last appended: 1.196, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
I20240502 21:27:08.278661  3963 raft_consensus.cc:1261] T dc17a741596a4cb59692b7f13cf254e7 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Refusing update from remote peer 68f426685b4c4c90bda61ef52177cc22: Log matching property violated. Preceding OpId in replica: term: 1 index: 196. Preceding OpId from leader: term: 1 index: 197. (index mismatch)
I20240502 21:27:08.280824  4813 consensus_queue.cc:1035] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 197, Last known committed idx: 196, Time since last communication: 0.000s
I20240502 21:27:08.299454  4772 raft_consensus.cc:2932] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Committing config change with OpId 1.197: config changed from index 155 to 197, VOTER c616208720db4b14a9219cae8f2b7d49 (127.27.196.67) evicted. New config: { opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:08.300583  3965 raft_consensus.cc:2932] T dc17a741596a4cb59692b7f13cf254e7 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Committing config change with OpId 1.197: config changed from index 155 to 197, VOTER c616208720db4b14a9219cae8f2b7d49 (127.27.196.67) evicted. New config: { opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:08.324803  2050 catalog_manager.cc:5484] T dc17a741596a4cb59692b7f13cf254e7 P 68f426685b4c4c90bda61ef52177cc22 reported cstate change: config changed from index 155 to 197, VOTER c616208720db4b14a9219cae8f2b7d49 (127.27.196.67) evicted. New cstate: current_term: 1 leader_uuid: "68f426685b4c4c90bda61ef52177cc22" committed_config { opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:08.348383  2447 tablet_service.cc:1514] Processing DeleteTablet for tablet dc17a741596a4cb59692b7f13cf254e7 with delete_type TABLET_DATA_TOMBSTONED (TS c616208720db4b14a9219cae8f2b7d49 not found in new config with opid_index 197) from {username='slave'} at 127.0.0.1:40204
I20240502 21:27:08.367206  4828 tablet_replica.cc:330] T dc17a741596a4cb59692b7f13cf254e7 P c616208720db4b14a9219cae8f2b7d49: stopping tablet replica
I20240502 21:27:08.420157  4828 raft_consensus.cc:2229] T dc17a741596a4cb59692b7f13cf254e7 P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:08.420876  4828 raft_consensus.cc:2258] T dc17a741596a4cb59692b7f13cf254e7 P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:08.447052  4828 ts_tablet_manager.cc:1875] T dc17a741596a4cb59692b7f13cf254e7 P c616208720db4b14a9219cae8f2b7d49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:08.470772  4828 ts_tablet_manager.cc:1888] T dc17a741596a4cb59692b7f13cf254e7 P c616208720db4b14a9219cae8f2b7d49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.196
I20240502 21:27:08.471076  4828 log.cc:1196] T dc17a741596a4cb59692b7f13cf254e7 P c616208720db4b14a9219cae8f2b7d49: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/wals/dc17a741596a4cb59692b7f13cf254e7
I20240502 21:27:08.472242  2037 catalog_manager.cc:4830] TS c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687): tablet dc17a741596a4cb59692b7f13cf254e7 (table rebalance_test_table_1 [id=2c42f3c4552d47169195e69073614d5b]) successfully deleted
W20240502 21:27:09.681670  2127 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.66:36253, user_credentials={real_user=slave}} blocked reactor thread for 47652.9us
W20240502 21:27:09.697430  2128 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.67:46687, user_credentials={real_user=slave}} blocked reactor thread for 48626us
W20240502 21:27:09.800694  4077 log.cc:924] Time spent T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
W20240502 21:27:09.888186  2263 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.67:46687, user_credentials={real_user=slave}} blocked reactor thread for 46425.9us
I20240502 21:27:10.018923  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:10.038754  3665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:10.154208  2308 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:27:10.156072  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:10.181713  3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:10.243292  2174 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 21:27:10.296828  2264 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.65:37025, user_credentials={real_user=slave}} blocked reactor thread for 49976.4us
I20240502 21:27:10.297957  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 21:27:11.020623  2127 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.66:36253, user_credentials={real_user=slave}} blocked reactor thread for 49068.1us
I20240502 21:27:11.130718  2192 consensus_queue.cc:237] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 194, Committed index: 194, Last appended: 1.194, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
I20240502 21:27:11.140376  3966 raft_consensus.cc:1261] T e93810d704674fc4ad4d96501fb5ea42 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Refusing update from remote peer d961985de7874e1990a19bb9b2da14c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 194. Preceding OpId from leader: term: 1 index: 195. (index mismatch)
I20240502 21:27:11.142016  4793 consensus_queue.cc:1035] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 195, Last known committed idx: 193, Time since last communication: 0.000s
I20240502 21:27:11.146299  4867 raft_consensus.cc:2932] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [term 1 LEADER]: Committing config change with OpId 1.195: config changed from index 122 to 195, VOTER c616208720db4b14a9219cae8f2b7d49 (127.27.196.67) evicted. New config: { opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:11.147413  3966 raft_consensus.cc:2932] T e93810d704674fc4ad4d96501fb5ea42 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Committing config change with OpId 1.195: config changed from index 122 to 195, VOTER c616208720db4b14a9219cae8f2b7d49 (127.27.196.67) evicted. New config: { opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:11.156057  2047 catalog_manager.cc:5484] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 reported cstate change: config changed from index 122 to 195, VOTER c616208720db4b14a9219cae8f2b7d49 (127.27.196.67) evicted. New cstate: current_term: 1 leader_uuid: "d961985de7874e1990a19bb9b2da14c2" committed_config { opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:11.322851  2447 tablet_service.cc:1514] Processing DeleteTablet for tablet e93810d704674fc4ad4d96501fb5ea42 with delete_type TABLET_DATA_TOMBSTONED (TS c616208720db4b14a9219cae8f2b7d49 not found in new config with opid_index 195) from {username='slave'} at 127.0.0.1:40204
I20240502 21:27:11.443346  4947 tablet_replica.cc:330] T e93810d704674fc4ad4d96501fb5ea42 P c616208720db4b14a9219cae8f2b7d49: stopping tablet replica
I20240502 21:27:11.449323  4947 raft_consensus.cc:2229] T e93810d704674fc4ad4d96501fb5ea42 P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:11.450505  4947 raft_consensus.cc:2258] T e93810d704674fc4ad4d96501fb5ea42 P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:11.453771  4947 ts_tablet_manager.cc:1875] T e93810d704674fc4ad4d96501fb5ea42 P c616208720db4b14a9219cae8f2b7d49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:11.471884  4947 ts_tablet_manager.cc:1888] T e93810d704674fc4ad4d96501fb5ea42 P c616208720db4b14a9219cae8f2b7d49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.194
I20240502 21:27:11.472167  4947 log.cc:1196] T e93810d704674fc4ad4d96501fb5ea42 P c616208720db4b14a9219cae8f2b7d49: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/wals/e93810d704674fc4ad4d96501fb5ea42
I20240502 21:27:11.474954  2037 catalog_manager.cc:4830] TS c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687): tablet e93810d704674fc4ad4d96501fb5ea42 (table rebalance_test_table_2 [id=dbf5b79f029b456d964098be120a38fd]) successfully deleted
I20240502 21:27:12.658387  3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:12.699750  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:12.773118  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:12.827136  3665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:12.848713  2308 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:27:13.004585  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:13.098156  2174 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 21:27:13.505918  2265 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.69:37783, user_credentials={real_user=slave}} blocked reactor thread for 68849.2us
I20240502 21:27:13.970418  2464 consensus_queue.cc:237] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 250, Committed index: 250, Last appended: 1.250, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
I20240502 21:27:13.975198  3963 raft_consensus.cc:1261] T 0306f5bfea0a49eca06554a40ba50d34 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Refusing update from remote peer c616208720db4b14a9219cae8f2b7d49: Log matching property violated. Preceding OpId in replica: term: 1 index: 250. Preceding OpId from leader: term: 1 index: 251. (index mismatch)
I20240502 21:27:13.976794  4966 consensus_queue.cc:1035] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I20240502 21:27:13.980914  5012 raft_consensus.cc:2932] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [term 1 LEADER]: Committing config change with OpId 1.251: config changed from index 124 to 251, VOTER d961985de7874e1990a19bb9b2da14c2 (127.27.196.65) evicted. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:13.985380  3963 raft_consensus.cc:2932] T 0306f5bfea0a49eca06554a40ba50d34 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Committing config change with OpId 1.251: config changed from index 124 to 251, VOTER d961985de7874e1990a19bb9b2da14c2 (127.27.196.65) evicted. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:13.991211  2050 catalog_manager.cc:5484] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 reported cstate change: config changed from index 124 to 251, VOTER d961985de7874e1990a19bb9b2da14c2 (127.27.196.65) evicted. New cstate: current_term: 1 leader_uuid: "c616208720db4b14a9219cae8f2b7d49" committed_config { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:14.161626  2174 tablet_service.cc:1514] Processing DeleteTablet for tablet 0306f5bfea0a49eca06554a40ba50d34 with delete_type TABLET_DATA_TOMBSTONED (TS d961985de7874e1990a19bb9b2da14c2 not found in new config with opid_index 251) from {username='slave'} at 127.0.0.1:41124
I20240502 21:27:14.162744  5054 tablet_replica.cc:330] T 0306f5bfea0a49eca06554a40ba50d34 P d961985de7874e1990a19bb9b2da14c2: stopping tablet replica
I20240502 21:27:14.165253  5054 raft_consensus.cc:2229] T 0306f5bfea0a49eca06554a40ba50d34 P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:14.165681  5054 raft_consensus.cc:2258] T 0306f5bfea0a49eca06554a40ba50d34 P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:14.169669  5054 ts_tablet_manager.cc:1875] T 0306f5bfea0a49eca06554a40ba50d34 P d961985de7874e1990a19bb9b2da14c2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:14.214047  5054 ts_tablet_manager.cc:1888] T 0306f5bfea0a49eca06554a40ba50d34 P d961985de7874e1990a19bb9b2da14c2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.250
I20240502 21:27:14.214377  5054 log.cc:1196] T 0306f5bfea0a49eca06554a40ba50d34 P d961985de7874e1990a19bb9b2da14c2: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/wals/0306f5bfea0a49eca06554a40ba50d34
I20240502 21:27:14.215703  2038 catalog_manager.cc:4830] TS d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025): tablet 0306f5bfea0a49eca06554a40ba50d34 (table rebalance_test_table_4 [id=69e9b99652e74772bfdd02eeaf9e1bd1]) successfully deleted
I20240502 21:27:15.338825  2308 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:27:15.414402  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:15.459650  3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:15.481948  3665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:15.509927  2174 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:15.586382  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:15.639999  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 21:27:15.666529  2265 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:46179, user_credentials={real_user=slave}} blocked reactor thread for 68712.4us
I20240502 21:27:16.429479  2191 consensus_queue.cc:237] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 320, Committed index: 320, Last appended: 1.321, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } }
I20240502 21:27:16.439947  3816 raft_consensus.cc:1261] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Refusing update from remote peer d961985de7874e1990a19bb9b2da14c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 321. Preceding OpId from leader: term: 1 index: 322. (index mismatch)
I20240502 21:27:16.441329  5116 consensus_queue.cc:1035] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 322, Last known committed idx: 320, Time since last communication: 0.000s
I20240502 21:27:16.449162  4867 raft_consensus.cc:2932] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [term 1 LEADER]: Committing config change with OpId 1.322: config changed from index 140 to 322, VOTER 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66) evicted. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } }
I20240502 21:27:16.461084  2050 catalog_manager.cc:5484] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 reported cstate change: config changed from index 140 to 322, VOTER 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66) evicted. New cstate: current_term: 1 leader_uuid: "d961985de7874e1990a19bb9b2da14c2" committed_config { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } }
I20240502 21:27:16.461005  3816 raft_consensus.cc:2932] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Committing config change with OpId 1.322: config changed from index 140 to 322, VOTER 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66) evicted. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } }
I20240502 21:27:16.534592  2308 tablet_service.cc:1514] Processing DeleteTablet for tablet 068c9a3114f648d1a183c9b90f4d365e with delete_type TABLET_DATA_TOMBSTONED (TS 68f426685b4c4c90bda61ef52177cc22 not found in new config with opid_index 322) from {username='slave'} at 127.0.0.1:35492
I20240502 21:27:16.576468  5143 tablet_replica.cc:330] T 068c9a3114f648d1a183c9b90f4d365e P 68f426685b4c4c90bda61ef52177cc22: stopping tablet replica
I20240502 21:27:16.581141  5143 raft_consensus.cc:2229] T 068c9a3114f648d1a183c9b90f4d365e P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:16.581584  5143 pending_rounds.cc:62] T 068c9a3114f648d1a183c9b90f4d365e P 68f426685b4c4c90bda61ef52177cc22: Trying to abort 1 pending ops.
I20240502 21:27:16.582026  5143 pending_rounds.cc:68] T 068c9a3114f648d1a183c9b90f4d365e P 68f426685b4c4c90bda61ef52177cc22: Aborting op as it isn't in flight: id { term: 1 index: 321 } timestamp: 7023350728357298176 op_type: WRITE_OP write_request { tablet_id: "068c9a3114f648d1a183c9b90f4d365e" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000.\036\177#\362!\241V\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""cd89752cf6d786f1" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023350728251760640 authz_token { token_data: "\010\311\214\320\261\006\"3\n\005slave\022*\n 2c42f3c4552d47169195e69073614d5b\020\001\030\001 \001(\001" signature: "<redacted>"" \235\344\251S\"ol\004\216\334]IV\306\250\240\376\270:\016\007\347\370\035\330*n3\222\315\312G\371\037\372%\036\250i1\215\315\217t\326\214Vi\314b\233\325\275`\n\266\253\014\302\\\354\211S" signing_key_seq_num: 0 } } request_id { client_id: "7248f4e0afa74f4393a8686937ece65f" seq_no: 966 first_incomplete_seq_no: 966 attempt_no: 0 }
I20240502 21:27:16.583200  5143 raft_consensus.cc:2258] T 068c9a3114f648d1a183c9b90f4d365e P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:16.596822  5143 ts_tablet_manager.cc:1875] T 068c9a3114f648d1a183c9b90f4d365e P 68f426685b4c4c90bda61ef52177cc22: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:16.629177  5143 ts_tablet_manager.cc:1888] T 068c9a3114f648d1a183c9b90f4d365e P 68f426685b4c4c90bda61ef52177cc22: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.321
I20240502 21:27:16.631407  5143 log.cc:1196] T 068c9a3114f648d1a183c9b90f4d365e P 68f426685b4c4c90bda61ef52177cc22: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/wals/068c9a3114f648d1a183c9b90f4d365e
I20240502 21:27:16.639858  2036 catalog_manager.cc:4830] TS 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253): tablet 068c9a3114f648d1a183c9b90f4d365e (table rebalance_test_table_1 [id=2c42f3c4552d47169195e69073614d5b]) successfully deleted
W20240502 21:27:17.537418  2265 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:46179, user_credentials={real_user=slave}} blocked reactor thread for 49053.1us
I20240502 21:27:17.947688  2308 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:27:17.973843  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:17.979540  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:18.000416  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:18.005410  3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:18.011941  2174 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:18.016109  3665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 21:27:18.194558  2401 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.69:37783, user_credentials={real_user=slave}} blocked reactor thread for 58583.2us
I20240502 21:27:18.935686  2193 consensus_queue.cc:237] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 311, Committed index: 311, Last appended: 1.311, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } }
I20240502 21:27:18.947393  3550 raft_consensus.cc:1261] T 0264c436aafa422a9a54599e74e0221c P 13451be0f7134a2c89c0c9330177213e [term 1 FOLLOWER]: Refusing update from remote peer d961985de7874e1990a19bb9b2da14c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 311. Preceding OpId from leader: term: 1 index: 312. (index mismatch)
I20240502 21:27:18.949657  5214 consensus_queue.cc:1035] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 312, Last known committed idx: 311, Time since last communication: 0.000s
I20240502 21:27:18.956977  4539 raft_consensus.cc:2932] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2 [term 1 LEADER]: Committing config change with OpId 1.312: config changed from index 89 to 312, VOTER c616208720db4b14a9219cae8f2b7d49 (127.27.196.67) evicted. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } }
I20240502 21:27:18.962977  3550 raft_consensus.cc:2932] T 0264c436aafa422a9a54599e74e0221c P 13451be0f7134a2c89c0c9330177213e [term 1 FOLLOWER]: Committing config change with OpId 1.312: config changed from index 89 to 312, VOTER c616208720db4b14a9219cae8f2b7d49 (127.27.196.67) evicted. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } }
I20240502 21:27:19.004858  2046 catalog_manager.cc:5484] T 0264c436aafa422a9a54599e74e0221c P d961985de7874e1990a19bb9b2da14c2 reported cstate change: config changed from index 89 to 312, VOTER c616208720db4b14a9219cae8f2b7d49 (127.27.196.67) evicted. New cstate: current_term: 1 leader_uuid: "d961985de7874e1990a19bb9b2da14c2" committed_config { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } }
I20240502 21:27:19.229679  2447 tablet_service.cc:1514] Processing DeleteTablet for tablet 0264c436aafa422a9a54599e74e0221c with delete_type TABLET_DATA_TOMBSTONED (TS c616208720db4b14a9219cae8f2b7d49 not found in new config with opid_index 312) from {username='slave'} at 127.0.0.1:40204
I20240502 21:27:19.286844  5255 tablet_replica.cc:330] T 0264c436aafa422a9a54599e74e0221c P c616208720db4b14a9219cae8f2b7d49: stopping tablet replica
I20240502 21:27:19.287338  5255 raft_consensus.cc:2229] T 0264c436aafa422a9a54599e74e0221c P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:19.287751  5255 raft_consensus.cc:2258] T 0264c436aafa422a9a54599e74e0221c P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:19.363572  5255 ts_tablet_manager.cc:1875] T 0264c436aafa422a9a54599e74e0221c P c616208720db4b14a9219cae8f2b7d49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:19.391141  5255 ts_tablet_manager.cc:1888] T 0264c436aafa422a9a54599e74e0221c P c616208720db4b14a9219cae8f2b7d49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.311
I20240502 21:27:19.391372  5255 log.cc:1196] T 0264c436aafa422a9a54599e74e0221c P c616208720db4b14a9219cae8f2b7d49: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/wals/0264c436aafa422a9a54599e74e0221c
I20240502 21:27:19.394587  2037 catalog_manager.cc:4830] TS c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687): tablet 0264c436aafa422a9a54599e74e0221c (table rebalance_test_table_4 [id=69e9b99652e74772bfdd02eeaf9e1bd1]) successfully deleted
I20240502 21:27:20.677078  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:20.704604  2308 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:27:20.709225  3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:20.711366  3665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:20.966084  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:20.967768  2174 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:21.015161  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:22.041296  2464 consensus_queue.cc:237] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 311, Committed index: 311, Last appended: 1.311, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } }
I20240502 21:27:22.043900  3683 raft_consensus.cc:1261] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Refusing update from remote peer c616208720db4b14a9219cae8f2b7d49: Log matching property violated. Preceding OpId in replica: term: 1 index: 311. Preceding OpId from leader: term: 1 index: 312. (index mismatch)
I20240502 21:27:22.044575  5210 consensus_queue.cc:1035] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 312, Last known committed idx: 311, Time since last communication: 0.000s
I20240502 21:27:22.048300  5210 raft_consensus.cc:2932] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [term 1 LEADER]: Committing config change with OpId 1.312: config changed from index 104 to 312, VOTER d961985de7874e1990a19bb9b2da14c2 (127.27.196.65) evicted. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } }
I20240502 21:27:22.054021  3683 raft_consensus.cc:2932] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Committing config change with OpId 1.312: config changed from index 104 to 312, VOTER d961985de7874e1990a19bb9b2da14c2 (127.27.196.65) evicted. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } }
I20240502 21:27:22.056129  2050 catalog_manager.cc:5484] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 reported cstate change: config changed from index 104 to 312, VOTER d961985de7874e1990a19bb9b2da14c2 (127.27.196.65) evicted. New cstate: current_term: 1 leader_uuid: "c616208720db4b14a9219cae8f2b7d49" committed_config { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } }
I20240502 21:27:22.244320  2174 tablet_service.cc:1514] Processing DeleteTablet for tablet d4392d14cf254b72bf506f9c6cfdd55d with delete_type TABLET_DATA_TOMBSTONED (TS d961985de7874e1990a19bb9b2da14c2 not found in new config with opid_index 312) from {username='slave'} at 127.0.0.1:41124
I20240502 21:27:22.245041  5353 tablet_replica.cc:330] T d4392d14cf254b72bf506f9c6cfdd55d P d961985de7874e1990a19bb9b2da14c2: stopping tablet replica
I20240502 21:27:22.245242  5353 raft_consensus.cc:2229] T d4392d14cf254b72bf506f9c6cfdd55d P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:22.245536  5353 raft_consensus.cc:2258] T d4392d14cf254b72bf506f9c6cfdd55d P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:22.247259  5353 ts_tablet_manager.cc:1875] T d4392d14cf254b72bf506f9c6cfdd55d P d961985de7874e1990a19bb9b2da14c2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:22.256706  5353 ts_tablet_manager.cc:1888] T d4392d14cf254b72bf506f9c6cfdd55d P d961985de7874e1990a19bb9b2da14c2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.311
I20240502 21:27:22.256924  5353 log.cc:1196] T d4392d14cf254b72bf506f9c6cfdd55d P d961985de7874e1990a19bb9b2da14c2: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/wals/d4392d14cf254b72bf506f9c6cfdd55d
I20240502 21:27:22.258636  2038 catalog_manager.cc:4830] TS d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025): tablet d4392d14cf254b72bf506f9c6cfdd55d (table rebalance_test_table_2 [id=dbf5b79f029b456d964098be120a38fd]) successfully deleted
I20240502 21:27:23.287477  2308 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:27:23.299945  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:23.303766  3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:23.335397  3665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:23.359510  2174 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:23.426968  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:23.499354  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:24.368269  2328 consensus_queue.cc:237] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 368, Committed index: 368, Last appended: 1.369, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } }
I20240502 21:27:24.374320  3683 raft_consensus.cc:1261] T be2cbcd79c1e40f49701271bb247f4d0 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Refusing update from remote peer 68f426685b4c4c90bda61ef52177cc22: Log matching property violated. Preceding OpId in replica: term: 1 index: 369. Preceding OpId from leader: term: 1 index: 370. (index mismatch)
I20240502 21:27:24.375437  4562 consensus_queue.cc:1035] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 370, Last known committed idx: 368, Time since last communication: 0.000s
I20240502 21:27:24.380088  5154 raft_consensus.cc:2932] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Committing config change with OpId 1.370: config changed from index 90 to 370, VOTER d961985de7874e1990a19bb9b2da14c2 (127.27.196.65) evicted. New config: { opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } }
I20240502 21:27:24.383903  3683 raft_consensus.cc:2932] T be2cbcd79c1e40f49701271bb247f4d0 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Committing config change with OpId 1.370: config changed from index 90 to 370, VOTER d961985de7874e1990a19bb9b2da14c2 (127.27.196.65) evicted. New config: { opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } }
I20240502 21:27:24.388334  2046 catalog_manager.cc:5484] T be2cbcd79c1e40f49701271bb247f4d0 P 68f426685b4c4c90bda61ef52177cc22 reported cstate change: config changed from index 90 to 370, VOTER d961985de7874e1990a19bb9b2da14c2 (127.27.196.65) evicted. New cstate: current_term: 1 leader_uuid: "68f426685b4c4c90bda61ef52177cc22" committed_config { opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } }
I20240502 21:27:24.558382  2174 tablet_service.cc:1514] Processing DeleteTablet for tablet be2cbcd79c1e40f49701271bb247f4d0 with delete_type TABLET_DATA_TOMBSTONED (TS d961985de7874e1990a19bb9b2da14c2 not found in new config with opid_index 370) from {username='slave'} at 127.0.0.1:41124
I20240502 21:27:24.726490  5444 tablet_replica.cc:330] T be2cbcd79c1e40f49701271bb247f4d0 P d961985de7874e1990a19bb9b2da14c2: stopping tablet replica
I20240502 21:27:24.726739  5444 raft_consensus.cc:2229] T be2cbcd79c1e40f49701271bb247f4d0 P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:24.728382  5444 pending_rounds.cc:62] T be2cbcd79c1e40f49701271bb247f4d0 P d961985de7874e1990a19bb9b2da14c2: Trying to abort 1 pending ops.
I20240502 21:27:24.728544  5444 pending_rounds.cc:68] T be2cbcd79c1e40f49701271bb247f4d0 P d961985de7874e1990a19bb9b2da14c2: Aborting op as it isn't in flight: id { term: 1 index: 369 } timestamp: 7023350760900370432 op_type: WRITE_OP write_request { tablet_id: "be2cbcd79c1e40f49701271bb247f4d0" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\202j\350\\\014E\322\323\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""87d325896e8a92c4" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023350760863555584 authz_token { token_data: "\010\311\214\320\261\006\"3\n\005slave\022*\n 69e9b99652e74772bfdd02eeaf9e1bd1\020\001\030\001 \001(\001" signature: "<redacted>""f\325\261\322qF\227\240\320_\263\233]\025mb\"S\201\330\007\303\236\364x7\212\313\333P\227\304\303\335\205\035\032(\217\000Z\025\256\355\251\221\305\340\373&S|\344EB}\256\3510\220\212\252\016a" signing_key_seq_num: 0 } } request_id { client_id: "40eba9a6e5f249138a8607147c2610dd" seq_no: 1128 first_incomplete_seq_no: 1128 attempt_no: 0 }
I20240502 21:27:24.733371  5444 raft_consensus.cc:2258] T be2cbcd79c1e40f49701271bb247f4d0 P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:24.736464  5444 ts_tablet_manager.cc:1875] T be2cbcd79c1e40f49701271bb247f4d0 P d961985de7874e1990a19bb9b2da14c2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:24.800235  5444 ts_tablet_manager.cc:1888] T be2cbcd79c1e40f49701271bb247f4d0 P d961985de7874e1990a19bb9b2da14c2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.369
I20240502 21:27:24.802613  5444 log.cc:1196] T be2cbcd79c1e40f49701271bb247f4d0 P d961985de7874e1990a19bb9b2da14c2: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/wals/be2cbcd79c1e40f49701271bb247f4d0
I20240502 21:27:24.904165  2038 catalog_manager.cc:4830] TS d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025): tablet be2cbcd79c1e40f49701271bb247f4d0 (table rebalance_test_table_4 [id=69e9b99652e74772bfdd02eeaf9e1bd1]) successfully deleted
I20240502 21:27:25.971275  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:25.993911  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:26.043774  3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:26.099606  2308 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:27:26.119745  3665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:26.125653  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 21:27:26.174777  2264 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.65:37025, user_credentials={real_user=slave}} blocked reactor thread for 60779.5us
I20240502 21:27:26.264771  2174 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:26.754876  2467 consensus_queue.cc:237] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 370, Committed index: 370, Last appended: 1.370, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
I20240502 21:27:26.757822  3964 raft_consensus.cc:1261] T d64bd5e7c22749078d82ad8da12d75dd P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Refusing update from remote peer c616208720db4b14a9219cae8f2b7d49: Log matching property violated. Preceding OpId in replica: term: 1 index: 370. Preceding OpId from leader: term: 1 index: 371. (index mismatch)
I20240502 21:27:26.758575  5210 consensus_queue.cc:1035] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 371, Last known committed idx: 370, Time since last communication: 0.000s
I20240502 21:27:26.764492  5310 raft_consensus.cc:2932] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49 [term 1 LEADER]: Committing config change with OpId 1.371: config changed from index 65 to 371, VOTER 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66) evicted. New config: { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:26.772145  3964 raft_consensus.cc:2932] T d64bd5e7c22749078d82ad8da12d75dd P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Committing config change with OpId 1.371: config changed from index 65 to 371, VOTER 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66) evicted. New config: { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:26.781240  2050 catalog_manager.cc:5484] T d64bd5e7c22749078d82ad8da12d75dd P c616208720db4b14a9219cae8f2b7d49 reported cstate change: config changed from index 65 to 371, VOTER 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66) evicted. New cstate: current_term: 1 leader_uuid: "c616208720db4b14a9219cae8f2b7d49" committed_config { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:26.830330  2308 tablet_service.cc:1514] Processing DeleteTablet for tablet d64bd5e7c22749078d82ad8da12d75dd with delete_type TABLET_DATA_TOMBSTONED (TS 68f426685b4c4c90bda61ef52177cc22 not found in new config with opid_index 371) from {username='slave'} at 127.0.0.1:35492
I20240502 21:27:26.852243  5543 tablet_replica.cc:330] T d64bd5e7c22749078d82ad8da12d75dd P 68f426685b4c4c90bda61ef52177cc22: stopping tablet replica
I20240502 21:27:26.852660  5543 raft_consensus.cc:2229] T d64bd5e7c22749078d82ad8da12d75dd P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:26.853078  5543 raft_consensus.cc:2258] T d64bd5e7c22749078d82ad8da12d75dd P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:26.857733  5543 ts_tablet_manager.cc:1875] T d64bd5e7c22749078d82ad8da12d75dd P 68f426685b4c4c90bda61ef52177cc22: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:26.876607  5543 ts_tablet_manager.cc:1888] T d64bd5e7c22749078d82ad8da12d75dd P 68f426685b4c4c90bda61ef52177cc22: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.370
I20240502 21:27:26.878090  5543 log.cc:1196] T d64bd5e7c22749078d82ad8da12d75dd P 68f426685b4c4c90bda61ef52177cc22: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/wals/d64bd5e7c22749078d82ad8da12d75dd
I20240502 21:27:26.884855  2036 catalog_manager.cc:4830] TS 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253): tablet d64bd5e7c22749078d82ad8da12d75dd (table rebalance_test_table_3 [id=72576758fa73429dba7089fdaa3f9ea3]) successfully deleted
I20240502 21:27:27.941473  3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:27.986873  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:28.071054  3665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:28.085538  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:28.097879  2308 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:27:28.119676  2174 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:28.225232  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:28.622562  2192 consensus_queue.cc:237] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 416, Committed index: 416, Last appended: 1.416, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 417 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } }
I20240502 21:27:28.624699  3552 raft_consensus.cc:1261] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e [term 1 FOLLOWER]: Refusing update from remote peer d961985de7874e1990a19bb9b2da14c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 416. Preceding OpId from leader: term: 1 index: 417. (index mismatch)
I20240502 21:27:28.625474  5572 consensus_queue.cc:1035] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 417, Last known committed idx: 416, Time since last communication: 0.000s
I20240502 21:27:28.628558  5572 raft_consensus.cc:2932] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [term 1 LEADER]: Committing config change with OpId 1.417: config changed from index 81 to 417, VOTER c616208720db4b14a9219cae8f2b7d49 (127.27.196.67) evicted. New config: { opid_index: 417 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } }
I20240502 21:27:28.634974  3552 raft_consensus.cc:2932] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e [term 1 FOLLOWER]: Committing config change with OpId 1.417: config changed from index 81 to 417, VOTER c616208720db4b14a9219cae8f2b7d49 (127.27.196.67) evicted. New config: { opid_index: 417 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } }
I20240502 21:27:28.641870  2047 catalog_manager.cc:5484] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e reported cstate change: config changed from index 81 to 417, VOTER c616208720db4b14a9219cae8f2b7d49 (127.27.196.67) evicted. New cstate: current_term: 1 leader_uuid: "d961985de7874e1990a19bb9b2da14c2" committed_config { opid_index: 417 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } }
I20240502 21:27:28.698107  2447 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e1c5a5cc57243d7bf6e212e3a32dfe3 with delete_type TABLET_DATA_TOMBSTONED (TS c616208720db4b14a9219cae8f2b7d49 not found in new config with opid_index 417) from {username='slave'} at 127.0.0.1:40204
I20240502 21:27:28.801072  5630 tablet_replica.cc:330] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P c616208720db4b14a9219cae8f2b7d49: stopping tablet replica
I20240502 21:27:28.801342  5630 raft_consensus.cc:2229] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:28.801628  5630 raft_consensus.cc:2258] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:28.805763  5630 ts_tablet_manager.cc:1875] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P c616208720db4b14a9219cae8f2b7d49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:28.823136  5630 ts_tablet_manager.cc:1888] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P c616208720db4b14a9219cae8f2b7d49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.416
I20240502 21:27:28.823912  5630 log.cc:1196] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P c616208720db4b14a9219cae8f2b7d49: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/wals/6e1c5a5cc57243d7bf6e212e3a32dfe3
I20240502 21:27:28.828823  2037 catalog_manager.cc:4830] TS c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687): tablet 6e1c5a5cc57243d7bf6e212e3a32dfe3 (table rebalance_test_table_0 [id=d7b10d11f7af4b32bf7fda57d6b60382]) successfully deleted
I20240502 21:27:29.374279  3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:29.379006  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:29.467432  3665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:29.559425  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:29.596916  2308 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:27:29.631867  2174 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:29.634200  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:30.207764  2329 consensus_queue.cc:237] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 382, Committed index: 382, Last appended: 1.382, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 383 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } }
I20240502 21:27:30.212102  3816 raft_consensus.cc:1261] T 6a38d2b4246e497fa91d2a0a8b127244 P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Refusing update from remote peer 68f426685b4c4c90bda61ef52177cc22: Log matching property violated. Preceding OpId in replica: term: 1 index: 382. Preceding OpId from leader: term: 1 index: 383. (index mismatch)
I20240502 21:27:30.213951  5445 consensus_queue.cc:1035] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 383, Last known committed idx: 382, Time since last communication: 0.000s
I20240502 21:27:30.228050  4562 raft_consensus.cc:2932] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Committing config change with OpId 1.383: config changed from index 66 to 383, VOTER d961985de7874e1990a19bb9b2da14c2 (127.27.196.65) evicted. New config: { opid_index: 383 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } }
I20240502 21:27:30.228780  3816 raft_consensus.cc:2932] T 6a38d2b4246e497fa91d2a0a8b127244 P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Committing config change with OpId 1.383: config changed from index 66 to 383, VOTER d961985de7874e1990a19bb9b2da14c2 (127.27.196.65) evicted. New config: { opid_index: 383 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } }
I20240502 21:27:30.239051  2046 catalog_manager.cc:5484] T 6a38d2b4246e497fa91d2a0a8b127244 P 68f426685b4c4c90bda61ef52177cc22 reported cstate change: config changed from index 66 to 383, VOTER d961985de7874e1990a19bb9b2da14c2 (127.27.196.65) evicted. New cstate: current_term: 1 leader_uuid: "68f426685b4c4c90bda61ef52177cc22" committed_config { opid_index: 383 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } }
I20240502 21:27:30.376825  2174 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a38d2b4246e497fa91d2a0a8b127244 with delete_type TABLET_DATA_TOMBSTONED (TS d961985de7874e1990a19bb9b2da14c2 not found in new config with opid_index 383) from {username='slave'} at 127.0.0.1:41124
I20240502 21:27:30.384615  5721 tablet_replica.cc:330] T 6a38d2b4246e497fa91d2a0a8b127244 P d961985de7874e1990a19bb9b2da14c2: stopping tablet replica
I20240502 21:27:30.384958  5721 raft_consensus.cc:2229] T 6a38d2b4246e497fa91d2a0a8b127244 P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:30.385288  5721 raft_consensus.cc:2258] T 6a38d2b4246e497fa91d2a0a8b127244 P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:30.393280  5721 ts_tablet_manager.cc:1875] T 6a38d2b4246e497fa91d2a0a8b127244 P d961985de7874e1990a19bb9b2da14c2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:30.400537  5721 ts_tablet_manager.cc:1888] T 6a38d2b4246e497fa91d2a0a8b127244 P d961985de7874e1990a19bb9b2da14c2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.382
I20240502 21:27:30.400808  5721 log.cc:1196] T 6a38d2b4246e497fa91d2a0a8b127244 P d961985de7874e1990a19bb9b2da14c2: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/wals/6a38d2b4246e497fa91d2a0a8b127244
I20240502 21:27:30.402380  2038 catalog_manager.cc:4830] TS d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025): tablet 6a38d2b4246e497fa91d2a0a8b127244 (table rebalance_test_table_2 [id=dbf5b79f029b456d964098be120a38fd]) successfully deleted
I20240502 21:27:31.206807  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:31.215003  3665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:31.255332  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:31.304952  2308 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:27:31.318035  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:31.341738  2174 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:31.362011  3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:27:32.022127  3481 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.71:39583, user_credentials={real_user=slave}} blocked reactor thread for 48834.7us
I20240502 21:27:32.507629  2329 consensus_queue.cc:237] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 483, Committed index: 483, Last appended: 1.484, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 485 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } }
I20240502 21:27:32.510192  3815 raft_consensus.cc:1261] T d7db9b73c0594885b383b059520414e9 P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Refusing update from remote peer 68f426685b4c4c90bda61ef52177cc22: Log matching property violated. Preceding OpId in replica: term: 1 index: 484. Preceding OpId from leader: term: 1 index: 485. (index mismatch)
I20240502 21:27:32.511719  4562 consensus_queue.cc:1035] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 485, Last known committed idx: 483, Time since last communication: 0.000s
I20240502 21:27:32.515036  4562 raft_consensus.cc:2932] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Committing config change with OpId 1.485: config changed from index 50 to 485, VOTER d961985de7874e1990a19bb9b2da14c2 (127.27.196.65) evicted. New config: { opid_index: 485 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } }
I20240502 21:27:32.522096  3815 raft_consensus.cc:2932] T d7db9b73c0594885b383b059520414e9 P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Committing config change with OpId 1.485: config changed from index 50 to 485, VOTER d961985de7874e1990a19bb9b2da14c2 (127.27.196.65) evicted. New config: { opid_index: 485 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } }
I20240502 21:27:32.534893  2047 catalog_manager.cc:5484] T d7db9b73c0594885b383b059520414e9 P 68f426685b4c4c90bda61ef52177cc22 reported cstate change: config changed from index 50 to 485, VOTER d961985de7874e1990a19bb9b2da14c2 (127.27.196.65) evicted. New cstate: current_term: 1 leader_uuid: "68f426685b4c4c90bda61ef52177cc22" committed_config { opid_index: 485 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } }
I20240502 21:27:32.797991  2174 tablet_service.cc:1514] Processing DeleteTablet for tablet d7db9b73c0594885b383b059520414e9 with delete_type TABLET_DATA_TOMBSTONED (TS d961985de7874e1990a19bb9b2da14c2 not found in new config with opid_index 485) from {username='slave'} at 127.0.0.1:41124
I20240502 21:27:32.847676  5832 tablet_replica.cc:330] T d7db9b73c0594885b383b059520414e9 P d961985de7874e1990a19bb9b2da14c2: stopping tablet replica
I20240502 21:27:32.848006  5832 raft_consensus.cc:2229] T d7db9b73c0594885b383b059520414e9 P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:32.851102  5832 pending_rounds.cc:62] T d7db9b73c0594885b383b059520414e9 P d961985de7874e1990a19bb9b2da14c2: Trying to abort 1 pending ops.
I20240502 21:27:32.851310  5832 pending_rounds.cc:68] T d7db9b73c0594885b383b059520414e9 P d961985de7874e1990a19bb9b2da14c2: Aborting op as it isn't in flight: id { term: 1 index: 484 } timestamp: 7023350794264207360 op_type: WRITE_OP write_request { tablet_id: "d7db9b73c0594885b383b059520414e9" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\007\217\230F\tu)\017\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""8907e332c9aa1a26" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023350794197540864 authz_token { token_data: "\010\311\214\320\261\006\"3\n\005slave\022*\n d7b10d11f7af4b32bf7fda57d6b60382\020\001\030\001 \001(\001" signature: "<redacted>""R\275_\201\254\266\277S\310E\364(\372T\227o\377\305\321H\341\205e\254\277\307%\032W\001\214X\300\250\252\237]7/6\3449@q\275G\246~\316\365\203c\267\247\273\367\201$\301\230,I\331\266" signing_key_seq_num: 0 } } request_id { client_id: "095f8c854fd74392a8a13e8bfd7348c3" seq_no: 1395 first_incomplete_seq_no: 1395 attempt_no: 0 }
I20240502 21:27:32.852279  5832 raft_consensus.cc:2258] T d7db9b73c0594885b383b059520414e9 P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:32.898783  5832 ts_tablet_manager.cc:1875] T d7db9b73c0594885b383b059520414e9 P d961985de7874e1990a19bb9b2da14c2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:32.970371  5832 ts_tablet_manager.cc:1888] T d7db9b73c0594885b383b059520414e9 P d961985de7874e1990a19bb9b2da14c2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.484
I20240502 21:27:32.970674  5832 log.cc:1196] T d7db9b73c0594885b383b059520414e9 P d961985de7874e1990a19bb9b2da14c2: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/wals/d7db9b73c0594885b383b059520414e9
I20240502 21:27:32.972034  2038 catalog_manager.cc:4830] TS d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025): tablet d7db9b73c0594885b383b059520414e9 (table rebalance_test_table_0 [id=d7b10d11f7af4b32bf7fda57d6b60382]) successfully deleted
I20240502 21:27:33.477582  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:33.493528  3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:33.511917  3665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:33.528060  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:33.575423  2308 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:27:33.596429  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:33.673568  2174 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:34.170161  2329 consensus_queue.cc:237] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 498, Committed index: 498, Last appended: 1.498, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 499 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } }
I20240502 21:27:34.178000  3685 raft_consensus.cc:1261] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Refusing update from remote peer 68f426685b4c4c90bda61ef52177cc22: Log matching property violated. Preceding OpId in replica: term: 1 index: 498. Preceding OpId from leader: term: 1 index: 499. (index mismatch)
I20240502 21:27:34.184698  5405 consensus_queue.cc:1035] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 499, Last known committed idx: 498, Time since last communication: 0.000s
I20240502 21:27:34.200109  5838 raft_consensus.cc:2932] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Committing config change with OpId 1.499: config changed from index 82 to 499, VOTER c616208720db4b14a9219cae8f2b7d49 (127.27.196.67) evicted. New config: { opid_index: 499 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } }
I20240502 21:27:34.204023  3685 raft_consensus.cc:2932] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Committing config change with OpId 1.499: config changed from index 82 to 499, VOTER c616208720db4b14a9219cae8f2b7d49 (127.27.196.67) evicted. New config: { opid_index: 499 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } }
I20240502 21:27:34.214994  2047 catalog_manager.cc:5484] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 reported cstate change: config changed from index 82 to 499, VOTER c616208720db4b14a9219cae8f2b7d49 (127.27.196.67) evicted. New cstate: current_term: 1 leader_uuid: "68f426685b4c4c90bda61ef52177cc22" committed_config { opid_index: 499 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } }
I20240502 21:27:34.384938  2447 tablet_service.cc:1514] Processing DeleteTablet for tablet 2779949906ff486da691638de7374050 with delete_type TABLET_DATA_TOMBSTONED (TS c616208720db4b14a9219cae8f2b7d49 not found in new config with opid_index 499) from {username='slave'} at 127.0.0.1:40204
I20240502 21:27:34.390018  5911 tablet_replica.cc:330] T 2779949906ff486da691638de7374050 P c616208720db4b14a9219cae8f2b7d49: stopping tablet replica
I20240502 21:27:34.391069  5911 raft_consensus.cc:2229] T 2779949906ff486da691638de7374050 P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:34.391539  5911 raft_consensus.cc:2258] T 2779949906ff486da691638de7374050 P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:34.394654  5911 ts_tablet_manager.cc:1875] T 2779949906ff486da691638de7374050 P c616208720db4b14a9219cae8f2b7d49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:34.409566  5911 ts_tablet_manager.cc:1888] T 2779949906ff486da691638de7374050 P c616208720db4b14a9219cae8f2b7d49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.498
I20240502 21:27:34.412052  5911 log.cc:1196] T 2779949906ff486da691638de7374050 P c616208720db4b14a9219cae8f2b7d49: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/wals/2779949906ff486da691638de7374050
I20240502 21:27:34.413219  2037 catalog_manager.cc:4830] TS c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687): tablet 2779949906ff486da691638de7374050 (table rebalance_test_table_3 [id=72576758fa73429dba7089fdaa3f9ea3]) successfully deleted
I20240502 21:27:35.651136  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:35.699534  3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:35.830132  2308 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:27:35.836951  3665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:35.853471  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:35.911465  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:35.918040  2174 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 21:27:36.710975  3481 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.66:36253, user_credentials={real_user=slave}} blocked reactor thread for 150431us
I20240502 21:27:37.015697  3550 consensus_queue.cc:237] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
I20240502 21:27:37.041199  3964 raft_consensus.cc:1261] T 5ae512d4d5874a909b1f4186c4b9785a P d306b26127e04e8c85a98edf1b41978a [term 2 FOLLOWER]: Refusing update from remote peer 13451be0f7134a2c89c0c9330177213e: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:27:37.042858  5789 consensus_queue.cc:1035] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:37.072489  5984 raft_consensus.cc:2932] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:37.073777  3966 raft_consensus.cc:2932] T 5ae512d4d5874a909b1f4186c4b9785a P d306b26127e04e8c85a98edf1b41978a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:37.081651  2045 catalog_manager.cc:5484] T 5ae512d4d5874a909b1f4186c4b9785a P 13451be0f7134a2c89c0c9330177213e reported cstate change: config changed from index 3 to 4, VOTER 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66) evicted. New cstate: current_term: 2 leader_uuid: "13451be0f7134a2c89c0c9330177213e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:37.151340  2308 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae512d4d5874a909b1f4186c4b9785a with delete_type TABLET_DATA_TOMBSTONED (TS 68f426685b4c4c90bda61ef52177cc22 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35492
I20240502 21:27:37.167470  6011 tablet_replica.cc:330] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22: stopping tablet replica
I20240502 21:27:37.167904  6011 raft_consensus.cc:2229] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:37.168248  6011 raft_consensus.cc:2258] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:37.173060  6011 ts_tablet_manager.cc:1875] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:37.186897  6011 ts_tablet_manager.cc:1888] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 21:27:37.187239  6011 log.cc:1196] T 5ae512d4d5874a909b1f4186c4b9785a P 68f426685b4c4c90bda61ef52177cc22: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/wals/5ae512d4d5874a909b1f4186c4b9785a
I20240502 21:27:37.190547  2036 catalog_manager.cc:4830] TS 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253): tablet 5ae512d4d5874a909b1f4186c4b9785a (table TestTable [id=4c82bfdabd1144f59d08b3d9f3d5cb56]) successfully deleted
I20240502 21:27:39.691092  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:39.717268  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:39.780143  2174 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:39.796725  2308 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:27:39.806027  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:39.811513  3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:39.844282  3665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:40.376904  2329 consensus_queue.cc:237] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 592, Committed index: 592, Last appended: 1.593, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 594 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } }
W20240502 21:27:40.380352  2265 consensus_peers.cc:485] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 -> Peer 1193f1d0d2c843139d98458cf4035eb8 (127.27.196.70:46179): Couldn't send request to peer 1193f1d0d2c843139d98458cf4035eb8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2779949906ff486da691638de7374050. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:40.387255  3685 raft_consensus.cc:1261] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Refusing update from remote peer 68f426685b4c4c90bda61ef52177cc22: Log matching property violated. Preceding OpId in replica: term: 1 index: 592. Preceding OpId from leader: term: 1 index: 594. (index mismatch)
I20240502 21:27:40.389672  6176 consensus_queue.cc:1035] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 594, Last known committed idx: 592, Time since last communication: 0.000s
I20240502 21:27:40.393397  3683 raft_consensus.cc:1203] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.592->[1.593-1.594]   Dedup: 1.593->[1.594-1.594]
I20240502 21:27:40.396922  6181 raft_consensus.cc:2932] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Committing config change with OpId 1.594: config changed from index 499 to 594, VOTER 1193f1d0d2c843139d98458cf4035eb8 (127.27.196.70) added. New config: { opid_index: 594 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } }
I20240502 21:27:40.417068  3683 raft_consensus.cc:2932] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Committing config change with OpId 1.594: config changed from index 499 to 594, VOTER 1193f1d0d2c843139d98458cf4035eb8 (127.27.196.70) added. New config: { opid_index: 594 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } }
I20240502 21:27:40.450069  2047 catalog_manager.cc:5484] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 reported cstate change: config changed from index 499 to 594, VOTER 1193f1d0d2c843139d98458cf4035eb8 (127.27.196.70) added. New cstate: current_term: 1 leader_uuid: "68f426685b4c4c90bda61ef52177cc22" committed_config { opid_index: 594 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } }
I20240502 21:27:40.905972  2329 consensus_queue.cc:237] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 703, Committed index: 703, Last appended: 1.703, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 704 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } }
I20240502 21:27:40.911255  2463 raft_consensus.cc:1261] T 99b8f0399d884e41b009da4c3f51fa93 P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Refusing update from remote peer 68f426685b4c4c90bda61ef52177cc22: Log matching property violated. Preceding OpId in replica: term: 1 index: 703. Preceding OpId from leader: term: 1 index: 704. (index mismatch)
I20240502 21:27:40.911970  6178 consensus_queue.cc:1035] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 704, Last known committed idx: 703, Time since last communication: 0.000s
I20240502 21:27:40.917240  6183 raft_consensus.cc:2932] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Committing config change with OpId 1.704: config changed from index -1 to 704, VOTER 4e8c16d1a1484a41a54b5ddc198becd3 (127.27.196.69) added. New config: { opid_index: 704 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } }
I20240502 21:27:40.925534  2463 raft_consensus.cc:2932] T 99b8f0399d884e41b009da4c3f51fa93 P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Committing config change with OpId 1.704: config changed from index -1 to 704, VOTER 4e8c16d1a1484a41a54b5ddc198becd3 (127.27.196.69) added. New config: { opid_index: 704 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } }
I20240502 21:27:40.938769  2047 catalog_manager.cc:5484] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 reported cstate change: config changed from index -1 to 704, VOTER 4e8c16d1a1484a41a54b5ddc198becd3 (127.27.196.69) added. New cstate: current_term: 1 leader_uuid: "68f426685b4c4c90bda61ef52177cc22" committed_config { opid_index: 704 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } }
W20240502 21:27:40.941584  2265 consensus_peers.cc:485] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 -> Peer 4e8c16d1a1484a41a54b5ddc198becd3 (127.27.196.69:37783): Couldn't send request to peer 4e8c16d1a1484a41a54b5ddc198becd3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99b8f0399d884e41b009da4c3f51fa93. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:40.991906  6221 ts_tablet_manager.cc:902] T 2779949906ff486da691638de7374050 P 1193f1d0d2c843139d98458cf4035eb8: Initiating tablet copy from peer 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253)
I20240502 21:27:41.045696  6221 tablet_copy_client.cc:308] T 2779949906ff486da691638de7374050 P 1193f1d0d2c843139d98458cf4035eb8: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:36253
I20240502 21:27:41.052404  2353 tablet_copy_service.cc:140] P 68f426685b4c4c90bda61ef52177cc22: Received BeginTabletCopySession request for tablet 2779949906ff486da691638de7374050 from peer 1193f1d0d2c843139d98458cf4035eb8 ({username='slave'} at 127.27.196.70:56283)
I20240502 21:27:41.052631  2353 tablet_copy_service.cc:161] P 68f426685b4c4c90bda61ef52177cc22: Beginning new tablet copy session on tablet 2779949906ff486da691638de7374050 from peer 1193f1d0d2c843139d98458cf4035eb8 at {username='slave'} at 127.27.196.70:56283: session id = 1193f1d0d2c843139d98458cf4035eb8-2779949906ff486da691638de7374050
I20240502 21:27:41.056830  2353 tablet_copy_source_session.cc:205] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:41.060263  6221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2779949906ff486da691638de7374050. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:41.069869  6221 tablet_copy_client.cc:783] T 2779949906ff486da691638de7374050 P 1193f1d0d2c843139d98458cf4035eb8: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:41.070782  6221 tablet_copy_client.cc:647] T 2779949906ff486da691638de7374050 P 1193f1d0d2c843139d98458cf4035eb8: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:41.134749  6221 tablet_copy_client.cc:523] T 2779949906ff486da691638de7374050 P 1193f1d0d2c843139d98458cf4035eb8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:41.146185  6221 tablet_bootstrap.cc:492] T 2779949906ff486da691638de7374050 P 1193f1d0d2c843139d98458cf4035eb8: Bootstrap starting.
I20240502 21:27:41.456094  2193 consensus_queue.cc:237] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 589, Committed index: 589, Last appended: 1.589, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 590 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
W20240502 21:27:41.460094  2127 consensus_peers.cc:485] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 -> Peer d306b26127e04e8c85a98edf1b41978a (127.27.196.71:39583): Couldn't send request to peer d306b26127e04e8c85a98edf1b41978a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e1c5a5cc57243d7bf6e212e3a32dfe3. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:41.460090  3552 raft_consensus.cc:1261] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e [term 1 FOLLOWER]: Refusing update from remote peer d961985de7874e1990a19bb9b2da14c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 589. Preceding OpId from leader: term: 1 index: 590. (index mismatch)
I20240502 21:27:41.461059  6200 consensus_queue.cc:1035] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 590, Last known committed idx: 589, Time since last communication: 0.000s
I20240502 21:27:41.463951  6200 raft_consensus.cc:2932] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [term 1 LEADER]: Committing config change with OpId 1.590: config changed from index 417 to 590, VOTER d306b26127e04e8c85a98edf1b41978a (127.27.196.71) added. New config: { opid_index: 590 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:41.465117  3552 raft_consensus.cc:2932] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e [term 1 FOLLOWER]: Committing config change with OpId 1.590: config changed from index 417 to 590, VOTER d306b26127e04e8c85a98edf1b41978a (127.27.196.71) added. New config: { opid_index: 590 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:41.484582  2047 catalog_manager.cc:5484] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e reported cstate change: config changed from index 417 to 590, VOTER d306b26127e04e8c85a98edf1b41978a (127.27.196.71) added. New cstate: current_term: 1 leader_uuid: "d961985de7874e1990a19bb9b2da14c2" committed_config { opid_index: 590 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:27:41.522018  6257 ts_tablet_manager.cc:902] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3: Initiating tablet copy from peer 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253)
I20240502 21:27:41.578109  6257 tablet_copy_client.cc:308] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:36253
I20240502 21:27:41.579285  2353 tablet_copy_service.cc:140] P 68f426685b4c4c90bda61ef52177cc22: Received BeginTabletCopySession request for tablet 99b8f0399d884e41b009da4c3f51fa93 from peer 4e8c16d1a1484a41a54b5ddc198becd3 ({username='slave'} at 127.27.196.69:44867)
I20240502 21:27:41.579488  2353 tablet_copy_service.cc:161] P 68f426685b4c4c90bda61ef52177cc22: Beginning new tablet copy session on tablet 99b8f0399d884e41b009da4c3f51fa93 from peer 4e8c16d1a1484a41a54b5ddc198becd3 at {username='slave'} at 127.27.196.69:44867: session id = 4e8c16d1a1484a41a54b5ddc198becd3-99b8f0399d884e41b009da4c3f51fa93
I20240502 21:27:41.582513  2353 tablet_copy_source_session.cc:205] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:41.585162  6257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99b8f0399d884e41b009da4c3f51fa93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:41.628407  6257 tablet_copy_client.cc:783] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:41.628868  6257 tablet_copy_client.cc:647] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:41.637488  6257 tablet_copy_client.cc:523] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:41.641506  6257 tablet_bootstrap.cc:492] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3: Bootstrap starting.
I20240502 21:27:42.019498  2193 consensus_queue.cc:237] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 517, Committed index: 517, Last appended: 1.518, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 519 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } }
I20240502 21:27:42.022852  3966 raft_consensus.cc:1261] T e93810d704674fc4ad4d96501fb5ea42 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Refusing update from remote peer d961985de7874e1990a19bb9b2da14c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 518. Preceding OpId from leader: term: 1 index: 519. (index mismatch)
I20240502 21:27:42.025882  6109 consensus_queue.cc:1035] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 519, Last known committed idx: 517, Time since last communication: 0.000s
I20240502 21:27:42.030623  6253 raft_consensus.cc:2932] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [term 1 LEADER]: Committing config change with OpId 1.519: config changed from index 195 to 519, VOTER 13451be0f7134a2c89c0c9330177213e (127.27.196.68) added. New config: { opid_index: 519 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } }
I20240502 21:27:42.039467  3964 raft_consensus.cc:2932] T e93810d704674fc4ad4d96501fb5ea42 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Committing config change with OpId 1.519: config changed from index 195 to 519, VOTER 13451be0f7134a2c89c0c9330177213e (127.27.196.68) added. New config: { opid_index: 519 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } }
W20240502 21:27:42.051983  2130 consensus_peers.cc:485] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 -> Peer 13451be0f7134a2c89c0c9330177213e (127.27.196.68:44575): Couldn't send request to peer 13451be0f7134a2c89c0c9330177213e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e93810d704674fc4ad4d96501fb5ea42. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:42.067807  6283 ts_tablet_manager.cc:902] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d306b26127e04e8c85a98edf1b41978a: Initiating tablet copy from peer d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025)
I20240502 21:27:42.073469  2045 catalog_manager.cc:5484] T e93810d704674fc4ad4d96501fb5ea42 P d306b26127e04e8c85a98edf1b41978a reported cstate change: config changed from index 195 to 519, VOTER 13451be0f7134a2c89c0c9330177213e (127.27.196.68) added. New cstate: current_term: 1 leader_uuid: "d961985de7874e1990a19bb9b2da14c2" committed_config { opid_index: 519 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } }
I20240502 21:27:42.381489  6283 tablet_copy_client.cc:308] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d306b26127e04e8c85a98edf1b41978a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:37025
I20240502 21:27:42.382768  2215 tablet_copy_service.cc:140] P d961985de7874e1990a19bb9b2da14c2: Received BeginTabletCopySession request for tablet 6e1c5a5cc57243d7bf6e212e3a32dfe3 from peer d306b26127e04e8c85a98edf1b41978a ({username='slave'} at 127.27.196.71:55737)
I20240502 21:27:42.382993  2215 tablet_copy_service.cc:161] P d961985de7874e1990a19bb9b2da14c2: Beginning new tablet copy session on tablet 6e1c5a5cc57243d7bf6e212e3a32dfe3 from peer d306b26127e04e8c85a98edf1b41978a at {username='slave'} at 127.27.196.71:55737: session id = d306b26127e04e8c85a98edf1b41978a-6e1c5a5cc57243d7bf6e212e3a32dfe3
I20240502 21:27:42.388464  2215 tablet_copy_source_session.cc:205] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:42.434715  6283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e1c5a5cc57243d7bf6e212e3a32dfe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:42.534101  6299 ts_tablet_manager.cc:902] T e93810d704674fc4ad4d96501fb5ea42 P 13451be0f7134a2c89c0c9330177213e: Initiating tablet copy from peer d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025)
I20240502 21:27:42.582383  6283 tablet_copy_client.cc:783] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d306b26127e04e8c85a98edf1b41978a: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:42.582828  6299 tablet_copy_client.cc:308] T e93810d704674fc4ad4d96501fb5ea42 P 13451be0f7134a2c89c0c9330177213e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:37025
I20240502 21:27:42.583904  2215 tablet_copy_service.cc:140] P d961985de7874e1990a19bb9b2da14c2: Received BeginTabletCopySession request for tablet e93810d704674fc4ad4d96501fb5ea42 from peer 13451be0f7134a2c89c0c9330177213e ({username='slave'} at 127.27.196.68:52125)
I20240502 21:27:42.584069  2215 tablet_copy_service.cc:161] P d961985de7874e1990a19bb9b2da14c2: Beginning new tablet copy session on tablet e93810d704674fc4ad4d96501fb5ea42 from peer 13451be0f7134a2c89c0c9330177213e at {username='slave'} at 127.27.196.68:52125: session id = 13451be0f7134a2c89c0c9330177213e-e93810d704674fc4ad4d96501fb5ea42
I20240502 21:27:42.604947  6283 tablet_copy_client.cc:647] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d306b26127e04e8c85a98edf1b41978a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:42.610754  6283 tablet_copy_client.cc:523] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d306b26127e04e8c85a98edf1b41978a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:42.614883  6283 tablet_bootstrap.cc:492] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d306b26127e04e8c85a98edf1b41978a: Bootstrap starting.
I20240502 21:27:42.632640  2215 tablet_copy_source_session.cc:205] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:42.634599  6299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93810d704674fc4ad4d96501fb5ea42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:42.642203  6299 tablet_copy_client.cc:783] T e93810d704674fc4ad4d96501fb5ea42 P 13451be0f7134a2c89c0c9330177213e: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:42.642634  6299 tablet_copy_client.cc:647] T e93810d704674fc4ad4d96501fb5ea42 P 13451be0f7134a2c89c0c9330177213e: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:42.646889  6299 tablet_copy_client.cc:523] T e93810d704674fc4ad4d96501fb5ea42 P 13451be0f7134a2c89c0c9330177213e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:42.656189  6299 tablet_bootstrap.cc:492] T e93810d704674fc4ad4d96501fb5ea42 P 13451be0f7134a2c89c0c9330177213e: Bootstrap starting.
I20240502 21:27:42.676105  6221 tablet_bootstrap.cc:492] T 2779949906ff486da691638de7374050 P 1193f1d0d2c843139d98458cf4035eb8: Bootstrap replayed 1/1 log segments. Stats: ops{read=607 overwritten=0 applied=606 ignored=0} inserts{seen=1289 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 21:27:42.676790  6221 tablet_bootstrap.cc:492] T 2779949906ff486da691638de7374050 P 1193f1d0d2c843139d98458cf4035eb8: Bootstrap complete.
I20240502 21:27:42.677322  6221 ts_tablet_manager.cc:1367] T 2779949906ff486da691638de7374050 P 1193f1d0d2c843139d98458cf4035eb8: Time spent bootstrapping tablet: real 1.531s	user 0.873s	sys 0.069s
I20240502 21:27:42.678522  6221 raft_consensus.cc:348] T 2779949906ff486da691638de7374050 P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 594 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } }
I20240502 21:27:42.679461  6221 raft_consensus.cc:729] T 2779949906ff486da691638de7374050 P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1193f1d0d2c843139d98458cf4035eb8, State: Initialized, Role: FOLLOWER
I20240502 21:27:42.679843  6221 consensus_queue.cc:260] T 2779949906ff486da691638de7374050 P 1193f1d0d2c843139d98458cf4035eb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 606, Last appended: 1.607, Last appended by leader: 607, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 594 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } }
I20240502 21:27:42.700579  6221 ts_tablet_manager.cc:1398] T 2779949906ff486da691638de7374050 P 1193f1d0d2c843139d98458cf4035eb8: Time spent starting tablet: real 0.022s	user 0.005s	sys 0.004s
I20240502 21:27:42.714025  2353 tablet_copy_service.cc:342] P 68f426685b4c4c90bda61ef52177cc22: Request end of tablet copy session 1193f1d0d2c843139d98458cf4035eb8-2779949906ff486da691638de7374050 received from {username='slave'} at 127.27.196.70:56283
I20240502 21:27:42.714200  2353 tablet_copy_service.cc:433] P 68f426685b4c4c90bda61ef52177cc22: ending tablet copy session 1193f1d0d2c843139d98458cf4035eb8-2779949906ff486da691638de7374050 on tablet 2779949906ff486da691638de7374050 with peer 1193f1d0d2c843139d98458cf4035eb8
I20240502 21:27:42.811903  3816 raft_consensus.cc:1203] T 2779949906ff486da691638de7374050 P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.593->[1.594-1.621]   Dedup: 1.607->[1.608-1.621]
W20240502 21:27:43.421798  2128 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.67:46687, user_credentials={real_user=slave}} blocked reactor thread for 91476.7us
W20240502 21:27:43.561828  6299 log.cc:924] Time spent T e93810d704674fc4ad4d96501fb5ea42 P 13451be0f7134a2c89c0c9330177213e: Append to log took a long time: real 0.077s	user 0.002s	sys 0.000s
W20240502 21:27:43.569711  2130 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.68:44575, user_credentials={real_user=slave}} blocked reactor thread for 45626.3us
W20240502 21:27:43.682461  6299 log.cc:924] Time spent T e93810d704674fc4ad4d96501fb5ea42 P 13451be0f7134a2c89c0c9330177213e: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
I20240502 21:27:43.747141  6257 tablet_bootstrap.cc:492] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=719 overwritten=0 applied=719 ignored=0} inserts{seen=1404 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:43.748095  6257 tablet_bootstrap.cc:492] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3: Bootstrap complete.
I20240502 21:27:43.748611  6257 ts_tablet_manager.cc:1367] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3: Time spent bootstrapping tablet: real 2.107s	user 0.975s	sys 0.087s
I20240502 21:27:43.749667  6257 raft_consensus.cc:348] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 704 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } }
I20240502 21:27:43.750046  6257 raft_consensus.cc:729] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e8c16d1a1484a41a54b5ddc198becd3, State: Initialized, Role: FOLLOWER
I20240502 21:27:43.753840  6257 consensus_queue.cc:260] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 719, Last appended: 1.719, Last appended by leader: 719, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 704 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } }
I20240502 21:27:43.768157  6257 ts_tablet_manager.cc:1398] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3: Time spent starting tablet: real 0.019s	user 0.004s	sys 0.000s
I20240502 21:27:43.774140  2353 tablet_copy_service.cc:342] P 68f426685b4c4c90bda61ef52177cc22: Request end of tablet copy session 4e8c16d1a1484a41a54b5ddc198becd3-99b8f0399d884e41b009da4c3f51fa93 received from {username='slave'} at 127.27.196.69:44867
I20240502 21:27:43.774327  2353 tablet_copy_service.cc:433] P 68f426685b4c4c90bda61ef52177cc22: ending tablet copy session 4e8c16d1a1484a41a54b5ddc198becd3-99b8f0399d884e41b009da4c3f51fa93 on tablet 99b8f0399d884e41b009da4c3f51fa93 with peer 4e8c16d1a1484a41a54b5ddc198becd3
I20240502 21:27:44.062537  3684 raft_consensus.cc:1203] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.703->[1.704-1.751]   Dedup: 1.719->[1.720-1.751]
W20240502 21:27:44.300415  2130 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.68:44575, user_credentials={real_user=slave}} blocked reactor thread for 49443.9us
W20240502 21:27:44.420667  2130 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:46179, user_credentials={real_user=slave}} blocked reactor thread for 90460.2us
W20240502 21:27:44.459911  2265 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:46179, user_credentials={real_user=slave}} blocked reactor thread for 58276.3us
W20240502 21:27:44.513489  2130 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:46179, user_credentials={real_user=slave}} blocked reactor thread for 45185.1us
W20240502 21:27:44.826421  2127 consensus_peers.cc:485] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 -> Peer d306b26127e04e8c85a98edf1b41978a (127.27.196.71:39583): Couldn't send request to peer d306b26127e04e8c85a98edf1b41978a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:44.836009  6283 tablet_bootstrap.cc:492] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d306b26127e04e8c85a98edf1b41978a: Bootstrap replayed 1/1 log segments. Stats: ops{read=601 overwritten=0 applied=601 ignored=0} inserts{seen=1284 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:44.836697  6283 tablet_bootstrap.cc:492] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d306b26127e04e8c85a98edf1b41978a: Bootstrap complete.
I20240502 21:27:44.837169  6283 ts_tablet_manager.cc:1367] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d306b26127e04e8c85a98edf1b41978a: Time spent bootstrapping tablet: real 2.222s	user 0.915s	sys 0.065s
I20240502 21:27:44.838636  6283 raft_consensus.cc:348] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 590 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
I20240502 21:27:44.839154  6283 raft_consensus.cc:729] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d306b26127e04e8c85a98edf1b41978a, State: Initialized, Role: FOLLOWER
I20240502 21:27:44.839774  6283 consensus_queue.cc:260] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d306b26127e04e8c85a98edf1b41978a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 601, Last appended: 1.601, Last appended by leader: 601, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 590 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
I20240502 21:27:44.847844  6283 ts_tablet_manager.cc:1398] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d306b26127e04e8c85a98edf1b41978a: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.007s
I20240502 21:27:44.849861  2215 tablet_copy_service.cc:342] P d961985de7874e1990a19bb9b2da14c2: Request end of tablet copy session d306b26127e04e8c85a98edf1b41978a-6e1c5a5cc57243d7bf6e212e3a32dfe3 received from {username='slave'} at 127.27.196.71:55737
I20240502 21:27:44.850054  2215 tablet_copy_service.cc:433] P d961985de7874e1990a19bb9b2da14c2: ending tablet copy session d306b26127e04e8c85a98edf1b41978a-6e1c5a5cc57243d7bf6e212e3a32dfe3 on tablet 6e1c5a5cc57243d7bf6e212e3a32dfe3 with peer d306b26127e04e8c85a98edf1b41978a
W20240502 21:27:44.877485  2130 consensus_peers.cc:485] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 -> Peer 13451be0f7134a2c89c0c9330177213e (127.27.196.68:44575): Couldn't send request to peer 13451be0f7134a2c89c0c9330177213e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:44.884110  6299 tablet_bootstrap.cc:492] T e93810d704674fc4ad4d96501fb5ea42 P 13451be0f7134a2c89c0c9330177213e: Bootstrap replayed 1/1 log segments. Stats: ops{read=526 overwritten=0 applied=526 ignored=0} inserts{seen=1155 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:44.884682  6299 tablet_bootstrap.cc:492] T e93810d704674fc4ad4d96501fb5ea42 P 13451be0f7134a2c89c0c9330177213e: Bootstrap complete.
I20240502 21:27:44.885092  6299 ts_tablet_manager.cc:1367] T e93810d704674fc4ad4d96501fb5ea42 P 13451be0f7134a2c89c0c9330177213e: Time spent bootstrapping tablet: real 2.229s	user 0.723s	sys 0.087s
I20240502 21:27:44.889497  6299 raft_consensus.cc:348] T e93810d704674fc4ad4d96501fb5ea42 P 13451be0f7134a2c89c0c9330177213e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 519 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } }
I20240502 21:27:44.890854  6299 raft_consensus.cc:729] T e93810d704674fc4ad4d96501fb5ea42 P 13451be0f7134a2c89c0c9330177213e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13451be0f7134a2c89c0c9330177213e, State: Initialized, Role: FOLLOWER
I20240502 21:27:44.891772  6299 consensus_queue.cc:260] T e93810d704674fc4ad4d96501fb5ea42 P 13451be0f7134a2c89c0c9330177213e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 526, Last appended: 1.526, Last appended by leader: 526, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 519 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } }
I20240502 21:27:44.894097  6299 ts_tablet_manager.cc:1398] T e93810d704674fc4ad4d96501fb5ea42 P 13451be0f7134a2c89c0c9330177213e: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240502 21:27:44.895522  2215 tablet_copy_service.cc:342] P d961985de7874e1990a19bb9b2da14c2: Request end of tablet copy session 13451be0f7134a2c89c0c9330177213e-e93810d704674fc4ad4d96501fb5ea42 received from {username='slave'} at 127.27.196.68:52125
I20240502 21:27:44.895668  2215 tablet_copy_service.cc:433] P d961985de7874e1990a19bb9b2da14c2: ending tablet copy session 13451be0f7134a2c89c0c9330177213e-e93810d704674fc4ad4d96501fb5ea42 on tablet e93810d704674fc4ad4d96501fb5ea42 with peer 13451be0f7134a2c89c0c9330177213e
I20240502 21:27:44.910517  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:44.996858  3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:45.012507  3665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:45.042148  2308 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:27:45.128558  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:45.218420  2174 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:45.248466  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:45.435492  3965 raft_consensus.cc:1203] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.589->[1.590-1.621]   Dedup: 1.601->[1.602-1.621]
I20240502 21:27:45.470435  3550 raft_consensus.cc:1203] T e93810d704674fc4ad4d96501fb5ea42 P 13451be0f7134a2c89c0c9330177213e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.518->[1.519-1.547]   Dedup: 1.526->[1.527-1.547]
I20240502 21:27:46.499388  2308 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:27:46.570318  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:46.596499  3665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:46.747030  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:46.781631  3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:46.813079  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:46.987434  2174 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 21:27:46.997108  2398 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.71:39583, user_credentials={real_user=slave}} blocked reactor thread for 58880.4us
W20240502 21:27:47.803416  2264 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.65:37025, user_credentials={real_user=slave}} blocked reactor thread for 50057.9us
I20240502 21:27:48.099916  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:48.118038  3665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:48.119705  3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:48.196758  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:48.204543  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:48.272888  2174 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:48.330665  2308 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:27:49.132627  2192 consensus_queue.cc:237] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 589, Committed index: 589, Last appended: 1.589, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 590 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } }
I20240502 21:27:49.134925  3551 raft_consensus.cc:1261] T e93810d704674fc4ad4d96501fb5ea42 P 13451be0f7134a2c89c0c9330177213e [term 1 FOLLOWER]: Refusing update from remote peer d961985de7874e1990a19bb9b2da14c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 589. Preceding OpId from leader: term: 1 index: 590. (index mismatch)
I20240502 21:27:49.136304  6433 consensus_queue.cc:1035] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 590, Last known committed idx: 589, Time since last communication: 0.000s
I20240502 21:27:49.143991  6352 raft_consensus.cc:2932] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 [term 1 LEADER]: Committing config change with OpId 1.590: config changed from index 519 to 590, VOTER d306b26127e04e8c85a98edf1b41978a (127.27.196.71) evicted. New config: { opid_index: 590 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } }
I20240502 21:27:49.153945  2047 catalog_manager.cc:5484] T e93810d704674fc4ad4d96501fb5ea42 P d961985de7874e1990a19bb9b2da14c2 reported cstate change: config changed from index 519 to 590, VOTER d306b26127e04e8c85a98edf1b41978a (127.27.196.71) evicted. New cstate: current_term: 1 leader_uuid: "d961985de7874e1990a19bb9b2da14c2" committed_config { opid_index: 590 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } }
I20240502 21:27:49.155182  3550 raft_consensus.cc:2932] T e93810d704674fc4ad4d96501fb5ea42 P 13451be0f7134a2c89c0c9330177213e [term 1 FOLLOWER]: Committing config change with OpId 1.590: config changed from index 519 to 590, VOTER d306b26127e04e8c85a98edf1b41978a (127.27.196.71) evicted. New config: { opid_index: 590 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } }
I20240502 21:27:49.279072  3944 tablet_service.cc:1514] Processing DeleteTablet for tablet e93810d704674fc4ad4d96501fb5ea42 with delete_type TABLET_DATA_TOMBSTONED (TS d306b26127e04e8c85a98edf1b41978a not found in new config with opid_index 590) from {username='slave'} at 127.0.0.1:42656
I20240502 21:27:49.289783  6569 tablet_replica.cc:330] T e93810d704674fc4ad4d96501fb5ea42 P d306b26127e04e8c85a98edf1b41978a: stopping tablet replica
I20240502 21:27:49.290951  6569 raft_consensus.cc:2229] T e93810d704674fc4ad4d96501fb5ea42 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:49.291677  6569 raft_consensus.cc:2258] T e93810d704674fc4ad4d96501fb5ea42 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:49.294502  6569 ts_tablet_manager.cc:1875] T e93810d704674fc4ad4d96501fb5ea42 P d306b26127e04e8c85a98edf1b41978a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:49.316632  6569 ts_tablet_manager.cc:1888] T e93810d704674fc4ad4d96501fb5ea42 P d306b26127e04e8c85a98edf1b41978a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.589
I20240502 21:27:49.320914  6569 log.cc:1196] T e93810d704674fc4ad4d96501fb5ea42 P d306b26127e04e8c85a98edf1b41978a: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal/wals/e93810d704674fc4ad4d96501fb5ea42
I20240502 21:27:49.327148  2036 catalog_manager.cc:4830] TS d306b26127e04e8c85a98edf1b41978a (127.27.196.71:39583): tablet e93810d704674fc4ad4d96501fb5ea42 (table rebalance_test_table_2 [id=dbf5b79f029b456d964098be120a38fd]) successfully deleted
I20240502 21:27:50.681488  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:50.714236  3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:50.714498  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:50.750698  2308 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:27:50.750834  3665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:50.779239  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:50.809660  2174 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 21:27:51.165721  3481 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.71:39583, user_credentials={real_user=slave}} blocked reactor thread for 122433us
I20240502 21:27:51.687925  2329 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "99b8f0399d884e41b009da4c3f51fa93"
dest_uuid: "68f426685b4c4c90bda61ef52177cc22"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:41038
I20240502 21:27:51.688252  2329 raft_consensus.cc:595] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Received request to transfer leadership
W20240502 21:27:51.694531  2287 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56496: Service unavailable: leader transfer in progress
W20240502 21:27:51.698746  2287 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56496: Service unavailable: leader transfer in progress
W20240502 21:27:51.706393  2287 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56496: Service unavailable: leader transfer in progress
W20240502 21:27:51.713888  2287 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56496: Service unavailable: leader transfer in progress
W20240502 21:27:51.724606  2287 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56496: Service unavailable: leader transfer in progress
W20240502 21:27:51.732852  2287 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56496: Service unavailable: leader transfer in progress
W20240502 21:27:51.744001  2287 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56496: Service unavailable: leader transfer in progress
W20240502 21:27:51.758891  2287 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56496: Service unavailable: leader transfer in progress
W20240502 21:27:51.773432  2287 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56496: Service unavailable: leader transfer in progress
W20240502 21:27:51.791687  2287 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56496: Service unavailable: leader transfer in progress
W20240502 21:27:51.806980  2287 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56496: Service unavailable: leader transfer in progress
W20240502 21:27:51.825367  2288 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56496: Service unavailable: leader transfer in progress
W20240502 21:27:51.842907  2288 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56496: Service unavailable: leader transfer in progress
W20240502 21:27:51.863545  2288 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56496: Service unavailable: leader transfer in progress
W20240502 21:27:51.884383  2288 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56496: Service unavailable: leader transfer in progress
W20240502 21:27:51.906988  2288 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56496: Service unavailable: leader transfer in progress
W20240502 21:27:51.929806  2288 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56496: Service unavailable: leader transfer in progress
W20240502 21:27:51.952800  2288 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56496: Service unavailable: leader transfer in progress
W20240502 21:27:51.975450  2288 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56496: Service unavailable: leader transfer in progress
W20240502 21:27:51.997458  2288 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56496: Service unavailable: leader transfer in progress
W20240502 21:27:52.023928  2288 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56496: Service unavailable: leader transfer in progress
W20240502 21:27:52.049970  2287 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56496: Service unavailable: leader transfer in progress
W20240502 21:27:52.078680  2287 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56496: Service unavailable: leader transfer in progress
I20240502 21:27:52.093614  6178 raft_consensus.cc:979] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22: : Instructing follower 4e8c16d1a1484a41a54b5ddc198becd3 to start an election
I20240502 21:27:52.093818  6639 raft_consensus.cc:1067] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Signalling peer 4e8c16d1a1484a41a54b5ddc198becd3 to start an election
I20240502 21:27:52.100471  3684 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "99b8f0399d884e41b009da4c3f51fa93"
dest_uuid: "4e8c16d1a1484a41a54b5ddc198becd3"
 from {username='slave'} at 127.27.196.66:53167
I20240502 21:27:52.100989  3684 raft_consensus.cc:482] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:52.101214  3684 raft_consensus.cc:3037] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:52.104964  3684 raft_consensus.cc:504] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 704 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } }
W20240502 21:27:52.108163  2288 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56496: Service unavailable: leader transfer in progress
I20240502 21:27:52.119657  2329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99b8f0399d884e41b009da4c3f51fa93" candidate_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 836 } } ignore_live_leader: true dest_uuid: "68f426685b4c4c90bda61ef52177cc22"
I20240502 21:27:52.120028  2329 raft_consensus.cc:3032] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:52.120141  2329 raft_consensus.cc:729] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 68f426685b4c4c90bda61ef52177cc22, State: Running, Role: LEADER
I20240502 21:27:52.120551  2329 consensus_queue.cc:260] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 836, Committed index: 836, Last appended: 1.836, Last appended by leader: 836, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 704 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } }
I20240502 21:27:52.121017  2329 raft_consensus.cc:3037] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:52.130185  2329 raft_consensus.cc:2454] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e8c16d1a1484a41a54b5ddc198becd3 in term 2.
I20240502 21:27:52.131419  3618 leader_election.cc:302] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e8c16d1a1484a41a54b5ddc198becd3, 68f426685b4c4c90bda61ef52177cc22; no voters: 
W20240502 21:27:52.139685  2288 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56496: Illegal state: replica 68f426685b4c4c90bda61ef52177cc22 is not leader of this config: current role FOLLOWER
I20240502 21:27:52.155812  3684 leader_election.cc:288] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3 [CANDIDATE]: Term 2 election: Requested vote from peers 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253), c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687)
I20240502 21:27:52.157171  2467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99b8f0399d884e41b009da4c3f51fa93" candidate_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 836 } } ignore_live_leader: true dest_uuid: "c616208720db4b14a9219cae8f2b7d49"
I20240502 21:27:52.157516  2467 raft_consensus.cc:3037] T 99b8f0399d884e41b009da4c3f51fa93 P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:52.162660  2467 raft_consensus.cc:2454] T 99b8f0399d884e41b009da4c3f51fa93 P c616208720db4b14a9219cae8f2b7d49 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e8c16d1a1484a41a54b5ddc198becd3 in term 2.
I20240502 21:27:52.180644  6672 raft_consensus.cc:2781] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 2 FOLLOWER]: Leader election won for term 2
W20240502 21:27:52.254556  2426 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60030: Illegal state: replica c616208720db4b14a9219cae8f2b7d49 is not leader of this config: current role FOLLOWER
W20240502 21:27:52.252700  3618 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.196.66:36253, user_credentials={real_user=slave}} blocked reactor thread for 121408us
I20240502 21:27:52.258356  6672 raft_consensus.cc:686] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 2 LEADER]: Becoming Leader. State: Replica: 4e8c16d1a1484a41a54b5ddc198becd3, State: Running, Role: LEADER
I20240502 21:27:52.258991  6672 consensus_queue.cc:237] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 836, Committed index: 836, Last appended: 1.836, Last appended by leader: 836, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 704 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } }
I20240502 21:27:52.272392  2046 catalog_manager.cc:5484] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3 reported cstate change: term changed from 1 to 2, leader changed from 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66) to 4e8c16d1a1484a41a54b5ddc198becd3 (127.27.196.69). New cstate: current_term: 2 leader_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" committed_config { opid_index: 704 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } }
W20240502 21:27:52.309743  2288 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56496: Illegal state: replica 68f426685b4c4c90bda61ef52177cc22 is not leader of this config: current role FOLLOWER
W20240502 21:27:52.343444  2424 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60030: Illegal state: replica c616208720db4b14a9219cae8f2b7d49 is not leader of this config: current role FOLLOWER
I20240502 21:27:52.449059  2329 raft_consensus.cc:1261] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [term 2 FOLLOWER]: Refusing update from remote peer 4e8c16d1a1484a41a54b5ddc198becd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 836. Preceding OpId from leader: term: 2 index: 838. (index mismatch)
I20240502 21:27:52.450592  6681 consensus_queue.cc:1035] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 837, Last known committed idx: 836, Time since last communication: 0.000s
I20240502 21:27:52.452333  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:52.453421  3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:52.456641  2467 raft_consensus.cc:1261] T 99b8f0399d884e41b009da4c3f51fa93 P c616208720db4b14a9219cae8f2b7d49 [term 2 FOLLOWER]: Refusing update from remote peer 4e8c16d1a1484a41a54b5ddc198becd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 836. Preceding OpId from leader: term: 2 index: 838. (index mismatch)
I20240502 21:27:52.461279  6681 consensus_queue.cc:1035] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 837, Last known committed idx: 836, Time since last communication: 0.000s
I20240502 21:27:52.462966  6666 mvcc.cc:205] Tried to move back new op lower bound from 7023350875942580224 to 7023350875214180352. Current Snapshot: MvccSnapshot[applied={T|T < 7023350875942580224}]
I20240502 21:27:52.471966  6638 mvcc.cc:205] Tried to move back new op lower bound from 7023350875942580224 to 7023350875214180352. Current Snapshot: MvccSnapshot[applied={T|T < 7023350875942580224}]
I20240502 21:27:52.510175  3665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:52.522855  2308 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:27:52.523818  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:52.553908  2174 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 21:27:52.554690  2265 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.69:37783, user_credentials={real_user=slave}} blocked reactor thread for 69870.1us
I20240502 21:27:52.615303  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 21:27:53.020537  2398 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.71:39583, user_credentials={real_user=slave}} blocked reactor thread for 77877.7us
I20240502 21:27:53.812851  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:53.815659  3665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:53.840178  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:53.851577  2308 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:27:53.864902  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:53.880852  2174 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:53.981263  3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:54.454875  2328 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2779949906ff486da691638de7374050"
dest_uuid: "68f426685b4c4c90bda61ef52177cc22"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:41054
I20240502 21:27:54.455154  2328 raft_consensus.cc:595] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:54.477807  6641 raft_consensus.cc:979] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22: : Instructing follower 4e8c16d1a1484a41a54b5ddc198becd3 to start an election
I20240502 21:27:54.478044  6641 raft_consensus.cc:1067] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Signalling peer 4e8c16d1a1484a41a54b5ddc198becd3 to start an election
I20240502 21:27:54.506608  3685 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2779949906ff486da691638de7374050"
dest_uuid: "4e8c16d1a1484a41a54b5ddc198becd3"
 from {username='slave'} at 127.27.196.66:53167
I20240502 21:27:54.506896  3685 raft_consensus.cc:482] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:54.506999  3685 raft_consensus.cc:3037] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:27:54.510026  2288 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56510: Service unavailable: leader transfer in progress
I20240502 21:27:54.511801  3685 raft_consensus.cc:504] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 594 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } }
I20240502 21:27:54.513252  2328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2779949906ff486da691638de7374050" candidate_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 740 } } ignore_live_leader: true dest_uuid: "68f426685b4c4c90bda61ef52177cc22"
I20240502 21:27:54.513522  2328 raft_consensus.cc:3032] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:54.513628  2328 raft_consensus.cc:729] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 68f426685b4c4c90bda61ef52177cc22, State: Running, Role: LEADER
I20240502 21:27:54.513931  2328 consensus_queue.cc:260] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 740, Committed index: 740, Last appended: 1.740, Last appended by leader: 740, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 594 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } }
I20240502 21:27:54.514362  2328 raft_consensus.cc:3037] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:54.516396  3685 leader_election.cc:288] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3 [CANDIDATE]: Term 2 election: Requested vote from peers 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253), 1193f1d0d2c843139d98458cf4035eb8 (127.27.196.70:46179)
I20240502 21:27:54.519393  2328 raft_consensus.cc:2454] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e8c16d1a1484a41a54b5ddc198becd3 in term 2.
I20240502 21:27:54.519996  3618 leader_election.cc:302] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e8c16d1a1484a41a54b5ddc198becd3, 68f426685b4c4c90bda61ef52177cc22; no voters: 
I20240502 21:27:54.523294  6723 raft_consensus.cc:2781] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:54.523501  6723 raft_consensus.cc:686] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 2 LEADER]: Becoming Leader. State: Replica: 4e8c16d1a1484a41a54b5ddc198becd3, State: Running, Role: LEADER
I20240502 21:27:54.523833  6723 consensus_queue.cc:237] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 740, Committed index: 740, Last appended: 1.740, Last appended by leader: 740, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 594 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } }
W20240502 21:27:54.532778  2288 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56510: Illegal state: replica 68f426685b4c4c90bda61ef52177cc22 is not leader of this config: current role FOLLOWER
W20240502 21:27:54.553890  2288 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56510: Illegal state: replica 68f426685b4c4c90bda61ef52177cc22 is not leader of this config: current role FOLLOWER
I20240502 21:27:54.663182  2047 catalog_manager.cc:5484] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3 reported cstate change: term changed from 1 to 2, leader changed from 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66) to 4e8c16d1a1484a41a54b5ddc198becd3 (127.27.196.69). New cstate: current_term: 2 leader_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" committed_config { opid_index: 594 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } }
I20240502 21:27:54.680020  2328 raft_consensus.cc:1261] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [term 2 FOLLOWER]: Refusing update from remote peer 4e8c16d1a1484a41a54b5ddc198becd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 740. Preceding OpId from leader: term: 2 index: 742. (index mismatch)
I20240502 21:27:54.680691  6672 consensus_queue.cc:1035] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 741, Last known committed idx: 740, Time since last communication: 0.000s
I20240502 21:27:54.693405  6764 mvcc.cc:205] Tried to move back new op lower bound from 7023350885075775488 to 7023350884452368384. Current Snapshot: MvccSnapshot[applied={T|T < 7023350885075775488}]
I20240502 21:27:54.826082  3815 raft_consensus.cc:3037] T 2779949906ff486da691638de7374050 P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:54.826961  3818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2779949906ff486da691638de7374050" candidate_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 740 } } ignore_live_leader: true dest_uuid: "1193f1d0d2c843139d98458cf4035eb8"
I20240502 21:27:54.832377  3815 raft_consensus.cc:1261] T 2779949906ff486da691638de7374050 P 1193f1d0d2c843139d98458cf4035eb8 [term 2 FOLLOWER]: Refusing update from remote peer 4e8c16d1a1484a41a54b5ddc198becd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 740. Preceding OpId from leader: term: 2 index: 742. (index mismatch)
I20240502 21:27:54.833343  6723 consensus_queue.cc:1035] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 741, Last known committed idx: 740, Time since last communication: 0.000s
W20240502 21:27:54.910387  2398 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.71:39583, user_credentials={real_user=slave}} blocked reactor thread for 46765.9us
W20240502 21:27:55.208469  3621 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:46179, user_credentials={real_user=slave}} blocked reactor thread for 87734.2us
I20240502 21:27:55.774597  3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:55.818845  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:55.822999  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:55.844877  3665 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:27:55.876566  2174 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:56.062165  2308 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:56.135803  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:56.829900  3685 consensus_queue.cc:237] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 773, Committed index: 773, Last appended: 2.773, Last appended by leader: 740, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 774 OBSOLETE_local: false peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } }
I20240502 21:27:56.839643  3815 raft_consensus.cc:1261] T 2779949906ff486da691638de7374050 P 1193f1d0d2c843139d98458cf4035eb8 [term 2 FOLLOWER]: Refusing update from remote peer 4e8c16d1a1484a41a54b5ddc198becd3: Log matching property violated. Preceding OpId in replica: term: 2 index: 772. Preceding OpId from leader: term: 2 index: 774. (index mismatch)
I20240502 21:27:56.840454  6897 consensus_queue.cc:1035] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 774, Last known committed idx: 772, Time since last communication: 0.000s
I20240502 21:27:56.848974  6897 raft_consensus.cc:2932] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 2 LEADER]: Committing config change with OpId 2.774: config changed from index 594 to 774, VOTER 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66) evicted. New config: { opid_index: 774 OBSOLETE_local: false peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } }
I20240502 21:27:56.849850  3816 raft_consensus.cc:2932] T 2779949906ff486da691638de7374050 P 1193f1d0d2c843139d98458cf4035eb8 [term 2 FOLLOWER]: Committing config change with OpId 2.774: config changed from index 594 to 774, VOTER 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66) evicted. New config: { opid_index: 774 OBSOLETE_local: false peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } }
I20240502 21:27:56.857152  2047 catalog_manager.cc:5484] T 2779949906ff486da691638de7374050 P 4e8c16d1a1484a41a54b5ddc198becd3 reported cstate change: config changed from index 594 to 774, VOTER 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66) evicted. New cstate: current_term: 2 leader_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" committed_config { opid_index: 774 OBSOLETE_local: false peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } }
I20240502 21:27:56.891245  3818 raft_consensus.cc:1203] T 2779949906ff486da691638de7374050 P 1193f1d0d2c843139d98458cf4035eb8 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.772->[2.773-2.773]   Dedup: 2.773->[]
I20240502 21:27:56.927279  2308 tablet_service.cc:1514] Processing DeleteTablet for tablet 2779949906ff486da691638de7374050 with delete_type TABLET_DATA_TOMBSTONED (TS 68f426685b4c4c90bda61ef52177cc22 not found in new config with opid_index 774) from {username='slave'} at 127.0.0.1:35492
I20240502 21:27:56.928215  6918 tablet_replica.cc:330] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22: stopping tablet replica
I20240502 21:27:56.931697  6918 raft_consensus.cc:2229] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:56.932096  6918 raft_consensus.cc:2258] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:56.942932  6918 ts_tablet_manager.cc:1875] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:56.955534  6918 ts_tablet_manager.cc:1888] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.773
I20240502 21:27:56.955744  6918 log.cc:1196] T 2779949906ff486da691638de7374050 P 68f426685b4c4c90bda61ef52177cc22: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/wals/2779949906ff486da691638de7374050
I20240502 21:27:56.957088  2036 catalog_manager.cc:4830] TS 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253): tablet 2779949906ff486da691638de7374050 (table rebalance_test_table_3 [id=72576758fa73429dba7089fdaa3f9ea3]) successfully deleted
W20240502 21:27:57.525525  2264 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.65:37025, user_credentials={real_user=slave}} blocked reactor thread for 114272us
I20240502 21:27:58.432760  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:58.846316  3530 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:27:58.936800  3621 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:46179, user_credentials={real_user=slave}} blocked reactor thread for 95625.6us
I20240502 21:27:59.019768  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:59.080447  2174 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:59.088323  2308 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:59.164162  3665 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:27:59.210309  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:59.592844  2192 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6e1c5a5cc57243d7bf6e212e3a32dfe3"
dest_uuid: "d961985de7874e1990a19bb9b2da14c2"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:39568
I20240502 21:27:59.593218  2192 raft_consensus.cc:595] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [term 1 LEADER]: Received request to transfer leadership
W20240502 21:27:59.858438  2153 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43850: Service unavailable: leader transfer in progress
W20240502 21:27:59.862350  2153 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43850: Service unavailable: leader transfer in progress
W20240502 21:27:59.870376  2153 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43850: Service unavailable: leader transfer in progress
W20240502 21:27:59.884995  2155 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43850: Service unavailable: leader transfer in progress
W20240502 21:27:59.900846  2153 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43850: Service unavailable: leader transfer in progress
W20240502 21:28:00.022635  2153 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43850: Service unavailable: leader transfer in progress
W20240502 21:28:00.064898  2153 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43850: Service unavailable: leader transfer in progress
I20240502 21:28:00.075402  6780 raft_consensus.cc:979] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2: : Instructing follower 13451be0f7134a2c89c0c9330177213e to start an election
I20240502 21:28:00.075662  6780 raft_consensus.cc:1067] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [term 1 LEADER]: Signalling peer 13451be0f7134a2c89c0c9330177213e to start an election
W20240502 21:28:00.076906  2153 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43850: Service unavailable: leader transfer in progress
I20240502 21:28:00.085239  3552 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6e1c5a5cc57243d7bf6e212e3a32dfe3"
dest_uuid: "13451be0f7134a2c89c0c9330177213e"
 from {username='slave'} at 127.27.196.65:47123
I20240502 21:28:00.085604  3552 raft_consensus.cc:482] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:28:00.085777  3552 raft_consensus.cc:3037] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:00.090655  3552 raft_consensus.cc:504] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 590 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
W20240502 21:28:00.091305  2153 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43850: Service unavailable: leader transfer in progress
I20240502 21:28:00.092618  3552 leader_election.cc:288] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e [CANDIDATE]: Term 2 election: Requested vote from peers d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025), d306b26127e04e8c85a98edf1b41978a (127.27.196.71:39583)
I20240502 21:28:00.095208  3965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e1c5a5cc57243d7bf6e212e3a32dfe3" candidate_uuid: "13451be0f7134a2c89c0c9330177213e" candidate_term: 2 candidate_status { last_received { term: 1 index: 835 } } ignore_live_leader: true dest_uuid: "d306b26127e04e8c85a98edf1b41978a"
I20240502 21:28:00.095444  2192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e1c5a5cc57243d7bf6e212e3a32dfe3" candidate_uuid: "13451be0f7134a2c89c0c9330177213e" candidate_term: 2 candidate_status { last_received { term: 1 index: 835 } } ignore_live_leader: true dest_uuid: "d961985de7874e1990a19bb9b2da14c2"
I20240502 21:28:00.095679  2192 raft_consensus.cc:3032] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:28:00.095768  2192 raft_consensus.cc:729] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d961985de7874e1990a19bb9b2da14c2, State: Running, Role: LEADER
I20240502 21:28:00.095753  3965 raft_consensus.cc:3037] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:00.096105  2192 consensus_queue.cc:260] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 835, Committed index: 835, Last appended: 1.835, Last appended by leader: 835, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 590 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
I20240502 21:28:00.096536  2192 raft_consensus.cc:3037] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:00.100606  2192 raft_consensus.cc:2454] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13451be0f7134a2c89c0c9330177213e in term 2.
I20240502 21:28:00.102185  3965 raft_consensus.cc:2454] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d306b26127e04e8c85a98edf1b41978a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13451be0f7134a2c89c0c9330177213e in term 2.
I20240502 21:28:00.102838  3483 leader_election.cc:302] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13451be0f7134a2c89c0c9330177213e, d961985de7874e1990a19bb9b2da14c2; no voters: 
I20240502 21:28:00.103291  7015 raft_consensus.cc:2781] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:28:00.104522  7015 raft_consensus.cc:686] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e [term 2 LEADER]: Becoming Leader. State: Replica: 13451be0f7134a2c89c0c9330177213e, State: Running, Role: LEADER
I20240502 21:28:00.104954  7015 consensus_queue.cc:237] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 835, Committed index: 835, Last appended: 1.835, Last appended by leader: 835, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 590 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
W20240502 21:28:00.105361  2153 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43850: Illegal state: replica d961985de7874e1990a19bb9b2da14c2 is not leader of this config: current role FOLLOWER
I20240502 21:28:00.108283  2050 catalog_manager.cc:5484] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e reported cstate change: term changed from 1 to 2, leader changed from d961985de7874e1990a19bb9b2da14c2 (127.27.196.65) to 13451be0f7134a2c89c0c9330177213e (127.27.196.68). New cstate: current_term: 2 leader_uuid: "13451be0f7134a2c89c0c9330177213e" committed_config { opid_index: 590 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:28:00.362248  3965 raft_consensus.cc:1261] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d306b26127e04e8c85a98edf1b41978a [term 2 FOLLOWER]: Refusing update from remote peer 13451be0f7134a2c89c0c9330177213e: Log matching property violated. Preceding OpId in replica: term: 1 index: 835. Preceding OpId from leader: term: 2 index: 837. (index mismatch)
I20240502 21:28:00.364930  2192 raft_consensus.cc:1261] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [term 2 FOLLOWER]: Refusing update from remote peer 13451be0f7134a2c89c0c9330177213e: Log matching property violated. Preceding OpId in replica: term: 1 index: 835. Preceding OpId from leader: term: 2 index: 837. (index mismatch)
I20240502 21:28:00.365279  7042 consensus_queue.cc:1035] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 836, Last known committed idx: 835, Time since last communication: 0.000s
I20240502 21:28:00.366783  7042 consensus_queue.cc:1035] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 836, Last known committed idx: 835, Time since last communication: 0.000s
I20240502 21:28:00.376147  6715 mvcc.cc:205] Tried to move back new op lower bound from 7023350908328574976 to 7023350907313377280. Current Snapshot: MvccSnapshot[applied={T|T < 7023350908328574976 or (T in {7023350908328574976})}]
I20240502 21:28:00.376974  6927 mvcc.cc:205] Tried to move back new op lower bound from 7023350908328574976 to 7023350907313377280. Current Snapshot: MvccSnapshot[applied={T|T < 7023350908328574976}]
I20240502 21:28:00.823333  2308 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:00.878106  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:00.887235  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:00.896917  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:00.904052  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:00.994346  3665 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:01.037169  2174 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:28:01.325017  3483 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.65:37025, user_credentials={real_user=slave}} blocked reactor thread for 98239.9us
W20240502 21:28:01.376268  2398 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.71:39583, user_credentials={real_user=slave}} blocked reactor thread for 53259.6us
W20240502 21:28:01.472059  2265 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:46179, user_credentials={real_user=slave}} blocked reactor thread for 81042.5us
I20240502 21:28:02.624029  3551 consensus_queue.cc:237] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 861, Committed index: 861, Last appended: 2.861, Last appended by leader: 835, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 862 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
I20240502 21:28:02.626142  3966 raft_consensus.cc:1261] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d306b26127e04e8c85a98edf1b41978a [term 2 FOLLOWER]: Refusing update from remote peer 13451be0f7134a2c89c0c9330177213e: Log matching property violated. Preceding OpId in replica: term: 2 index: 860. Preceding OpId from leader: term: 2 index: 862. (index mismatch)
I20240502 21:28:02.638263  7084 consensus_queue.cc:1035] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 862, Last known committed idx: 860, Time since last communication: 0.000s
I20240502 21:28:02.641209  3965 raft_consensus.cc:1203] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d306b26127e04e8c85a98edf1b41978a [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.860->[2.861-2.862]   Dedup: 2.861->[2.862-2.862]
I20240502 21:28:02.644486  7082 raft_consensus.cc:2932] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P 13451be0f7134a2c89c0c9330177213e [term 2 LEADER]: Committing config change with OpId 2.862: config changed from index 590 to 862, VOTER d961985de7874e1990a19bb9b2da14c2 (127.27.196.65) evicted. New config: { opid_index: 862 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:28:02.645409  3963 raft_consensus.cc:2932] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d306b26127e04e8c85a98edf1b41978a [term 2 FOLLOWER]: Committing config change with OpId 2.862: config changed from index 590 to 862, VOTER d961985de7874e1990a19bb9b2da14c2 (127.27.196.65) evicted. New config: { opid_index: 862 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:28:02.669860  2050 catalog_manager.cc:5484] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d306b26127e04e8c85a98edf1b41978a reported cstate change: config changed from index 590 to 862, VOTER d961985de7874e1990a19bb9b2da14c2 (127.27.196.65) evicted. New cstate: current_term: 2 leader_uuid: "13451be0f7134a2c89c0c9330177213e" committed_config { opid_index: 862 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } }
I20240502 21:28:02.703720  2174 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e1c5a5cc57243d7bf6e212e3a32dfe3 with delete_type TABLET_DATA_TOMBSTONED (TS d961985de7874e1990a19bb9b2da14c2 not found in new config with opid_index 862) from {username='slave'} at 127.0.0.1:41124
I20240502 21:28:02.711510  7138 tablet_replica.cc:330] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2: stopping tablet replica
I20240502 21:28:02.712349  7138 raft_consensus.cc:2229] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:02.713048  7138 raft_consensus.cc:2258] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:02.732021  7138 ts_tablet_manager.cc:1875] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:02.754729  7138 ts_tablet_manager.cc:1888] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.861
I20240502 21:28:02.755642  7138 log.cc:1196] T 6e1c5a5cc57243d7bf6e212e3a32dfe3 P d961985de7874e1990a19bb9b2da14c2: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/wals/6e1c5a5cc57243d7bf6e212e3a32dfe3
I20240502 21:28:02.760036  2038 catalog_manager.cc:4830] TS d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025): tablet 6e1c5a5cc57243d7bf6e212e3a32dfe3 (table rebalance_test_table_0 [id=d7b10d11f7af4b32bf7fda57d6b60382]) successfully deleted
I20240502 21:28:03.440725  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:03.529807  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:03.563614  2174 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:03.598769  2308 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:03.624385  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:03.633242  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:03.749228  3665 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:04.501127  3685 consensus_queue.cc:237] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1006, Committed index: 1006, Last appended: 2.1006, Last appended by leader: 836, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1007 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } }
I20240502 21:28:04.503944  2463 raft_consensus.cc:1261] T 99b8f0399d884e41b009da4c3f51fa93 P c616208720db4b14a9219cae8f2b7d49 [term 2 FOLLOWER]: Refusing update from remote peer 4e8c16d1a1484a41a54b5ddc198becd3: Log matching property violated. Preceding OpId in replica: term: 2 index: 1006. Preceding OpId from leader: term: 2 index: 1007. (index mismatch)
I20240502 21:28:04.505519  7228 consensus_queue.cc:1035] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1007, Last known committed idx: 1006, Time since last communication: 0.000s
I20240502 21:28:04.512054  7228 raft_consensus.cc:2932] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 2 LEADER]: Committing config change with OpId 2.1007: config changed from index 704 to 1007, VOTER 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66) evicted. New config: { opid_index: 1007 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } }
I20240502 21:28:04.517304  2463 raft_consensus.cc:2932] T 99b8f0399d884e41b009da4c3f51fa93 P c616208720db4b14a9219cae8f2b7d49 [term 2 FOLLOWER]: Committing config change with OpId 2.1007: config changed from index 704 to 1007, VOTER 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66) evicted. New config: { opid_index: 1007 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } }
I20240502 21:28:04.529556  2045 catalog_manager.cc:5484] T 99b8f0399d884e41b009da4c3f51fa93 P c616208720db4b14a9219cae8f2b7d49 reported cstate change: config changed from index 704 to 1007, VOTER 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66) evicted. New cstate: current_term: 2 leader_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" committed_config { opid_index: 1007 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } }
W20240502 21:28:04.677034  2130 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.68:44575, user_credentials={real_user=slave}} blocked reactor thread for 76430.7us
I20240502 21:28:04.698736  2308 tablet_service.cc:1514] Processing DeleteTablet for tablet 99b8f0399d884e41b009da4c3f51fa93 with delete_type TABLET_DATA_TOMBSTONED (TS 68f426685b4c4c90bda61ef52177cc22 not found in new config with opid_index 1007) from {username='slave'} at 127.0.0.1:35492
I20240502 21:28:04.703698  7247 tablet_replica.cc:330] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22: stopping tablet replica
I20240502 21:28:04.707551  7247 raft_consensus.cc:2229] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:04.708622  7247 pending_rounds.cc:62] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22: Trying to abort 1 pending ops.
I20240502 21:28:04.708845  7247 pending_rounds.cc:68] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22: Aborting op as it isn't in flight: id { term: 2 index: 1006 } timestamp: 7023350925262708736 op_type: WRITE_OP write_request { tablet_id: "99b8f0399d884e41b009da4c3f51fa93" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\035\232\373$=\000\342\372\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""fafbfaf16a3004ef" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023350925090443264 authz_token { token_data: "\010\311\214\320\261\006\"3\n\005slave\022*\n 2c42f3c4552d47169195e69073614d5b\020\001\030\001 \001(\001" signature: "<redacted>"" \235\344\251S\"ol\004\216\334]IV\306\250\240\376\270:\016\007\347\370\035\330*n3\222\315\312G\371\037\372%\036\250i1\215\315\217t\326\214Vi\314b\233\325\275`\n\266\253\014\302\\\354\211S" signing_key_seq_num: 0 } } request_id { client_id: "7248f4e0afa74f4393a8686937ece65f" seq_no: 2934 first_incomplete_seq_no: 2934 attempt_no: 0 }
I20240502 21:28:04.710204  7247 raft_consensus.cc:2258] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:04.721036  7247 ts_tablet_manager.cc:1875] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:04.730098  7247 ts_tablet_manager.cc:1888] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1006
I20240502 21:28:04.747270  7247 log.cc:1196] T 99b8f0399d884e41b009da4c3f51fa93 P 68f426685b4c4c90bda61ef52177cc22: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/wals/99b8f0399d884e41b009da4c3f51fa93
I20240502 21:28:04.748461  2036 catalog_manager.cc:4830] TS 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253): tablet 99b8f0399d884e41b009da4c3f51fa93 (table rebalance_test_table_1 [id=2c42f3c4552d47169195e69073614d5b]) successfully deleted
I20240502 21:28:06.347738  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:06.363603  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:06.384451  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:06.405021  2308 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:06.405150  3665 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:06.464663  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:06.650978  2174 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:28:07.062871  2128 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.67:46687, user_credentials={real_user=slave}} blocked reactor thread for 55735.7us
I20240502 21:28:08.002250  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:08.025357  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:08.137888  2308 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:08.182323  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:08.226186  2174 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:08.251803  3665 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:28:08.261420  2130 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.68:44575, user_credentials={real_user=slave}} blocked reactor thread for 57973.3us
I20240502 21:28:08.310611  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:11.052448  2308 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:11.064996  2174 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:11.068145  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:11.098651  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:11.122892  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:11.147343  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:11.236617  3665 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:12.740135  2308 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:12.776959  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:12.783110  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:12.815032  2174 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:12.823828  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:12.861138  3665 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:12.918617  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 21:28:13.053269  2130 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.68:44575, user_credentials={real_user=slave}} blocked reactor thread for 104243us
I20240502 21:28:13.818794  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:13.840453  3665 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:13.841147  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:13.860078  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:13.872836  2308 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:13.876096  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:13.879179  2174 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 b87cabdf7aa145babe0be5701e5c3815 | 127.27.196.126:43629 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                Value                                                                                 |         Tags          |         Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 allow_unsafe_replication_factor          | true                                                                                                                                                                 | runtime,unsafe        | all 1 server(s) checked
 ipki_ca_key_size                         | 768                                                                                                                                                                  | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                  | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                 | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                    | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | false                                                                                                                                                                | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                 | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                 | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                   | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                  | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |        Value        |         Master
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.27.196.84:40533 | all 1 server(s) checked
 time_source         | builtin             | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 1193f1d0d2c843139d98458cf4035eb8 | 127.27.196.70:46179 | HEALTHY | <none>   |       0        |       0
 13451be0f7134a2c89c0c9330177213e | 127.27.196.68:44575 | HEALTHY | <none>   |       2        |       0
 4e8c16d1a1484a41a54b5ddc198becd3 | 127.27.196.69:37783 | HEALTHY | <none>   |       2        |       0
 68f426685b4c4c90bda61ef52177cc22 | 127.27.196.66:36253 | HEALTHY | <none>   |       5        |       0
 c616208720db4b14a9219cae8f2b7d49 | 127.27.196.67:46687 | HEALTHY | <none>   |       3        |       0
 d306b26127e04e8c85a98edf1b41978a | 127.27.196.71:39583 | HEALTHY | <none>   |       0        |       0
 d961985de7874e1990a19bb9b2da14c2 | 127.27.196.65:37025 | HEALTHY | <none>   |       4        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       7

Unusual flags for Tablet Server:
                   Flag                   |                                                                              Value                                                                               |         Tags          |      Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                              | experimental          | all 7 server(s) checked
 local_ip_for_outbound_sockets            | 127.27.196.65                                                                                                                                                    | experimental          | 127.27.196.65:37025
 local_ip_for_outbound_sockets            | 127.27.196.66                                                                                                                                                    | experimental          | 127.27.196.66:36253
 local_ip_for_outbound_sockets            | 127.27.196.67                                                                                                                                                    | experimental          | 127.27.196.67:46687
 local_ip_for_outbound_sockets            | 127.27.196.68                                                                                                                                                    | experimental          | 127.27.196.68:44575
 local_ip_for_outbound_sockets            | 127.27.196.69                                                                                                                                                    | experimental          | 127.27.196.69:37783
 local_ip_for_outbound_sockets            | 127.27.196.70                                                                                                                                                    | experimental          | 127.27.196.70:46179
 local_ip_for_outbound_sockets            | 127.27.196.71                                                                                                                                                    | experimental          | 127.27.196.71:39583
 never_fsync                              | true                                                                                                                                                             | unsafe,advanced       | all 7 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                | unsafe,hidden         | all 7 server(s) checked
 raft_prepare_replacement_before_eviction | false                                                                                                                                                            | experimental,advanced | all 7 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                             | unsafe                | all 7 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                               | hidden                | all 7 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.27.196.65:37025
 server_dump_info_path                    | /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.27.196.66:36253
 server_dump_info_path                    | /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.27.196.67:46687
 server_dump_info_path                    | /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.27.196.68:44575
 server_dump_info_path                    | /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden                | 127.27.196.69:37783
 server_dump_info_path                    | /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden                | 127.27.196.70:46179
 server_dump_info_path                    | /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden                | 127.27.196.71:39583

Flags of checked categories for Tablet Server:
        Flag         |        Value        |      Tablet Server
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.27.196.84:40533 | all 7 server(s) checked
 time_source         | builtin             | all 7 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 8 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 2  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_4 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 4
 First Quartile | 4
 Median         | 5
 Third Quartile | 5
 Maximum        | 5

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 7
 Tables         | 6
 Tablets        | 16
 Replicas       | 32

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240502 21:28:13.997750 28433 log_verifier.cc:126] Checking tablet 0264c436aafa422a9a54599e74e0221c
I20240502 21:28:14.368906 28433 log_verifier.cc:177] Verified matching terms for 927 ops in tablet 0264c436aafa422a9a54599e74e0221c
I20240502 21:28:14.370378 28433 log_verifier.cc:126] Checking tablet 0306f5bfea0a49eca06554a40ba50d34
I20240502 21:28:14.763839 28433 log_verifier.cc:177] Verified matching terms for 968 ops in tablet 0306f5bfea0a49eca06554a40ba50d34
I20240502 21:28:14.765285 28433 log_verifier.cc:126] Checking tablet 068c9a3114f648d1a183c9b90f4d365e
I20240502 21:28:15.170451 28433 log_verifier.cc:177] Verified matching terms for 1100 ops in tablet 068c9a3114f648d1a183c9b90f4d365e
I20240502 21:28:15.171998 28433 log_verifier.cc:126] Checking tablet 2779949906ff486da691638de7374050
I20240502 21:28:15.527665 28433 log_verifier.cc:177] Verified matching terms for 998 ops in tablet 2779949906ff486da691638de7374050
I20240502 21:28:15.529577 28433 log_verifier.cc:126] Checking tablet 3a6da907ad0447b3ad2f66f50cd7289d
I20240502 21:28:15.873173 28433 log_verifier.cc:177] Verified matching terms for 922 ops in tablet 3a6da907ad0447b3ad2f66f50cd7289d
I20240502 21:28:15.874217 28433 log_verifier.cc:126] Checking tablet 5ae512d4d5874a909b1f4186c4b9785a
I20240502 21:28:15.892040 28433 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 5ae512d4d5874a909b1f4186c4b9785a
I20240502 21:28:15.892215 28433 log_verifier.cc:126] Checking tablet 6a38d2b4246e497fa91d2a0a8b127244
I20240502 21:28:16.217373 28433 log_verifier.cc:177] Verified matching terms for 908 ops in tablet 6a38d2b4246e497fa91d2a0a8b127244
I20240502 21:28:16.218758 28433 log_verifier.cc:126] Checking tablet 6e1c5a5cc57243d7bf6e212e3a32dfe3
I20240502 21:28:16.585968 28433 log_verifier.cc:177] Verified matching terms for 1013 ops in tablet 6e1c5a5cc57243d7bf6e212e3a32dfe3
I20240502 21:28:16.588413 28433 log_verifier.cc:126] Checking tablet 99b8f0399d884e41b009da4c3f51fa93
I20240502 21:28:17.003257 28433 log_verifier.cc:177] Verified matching terms for 1132 ops in tablet 99b8f0399d884e41b009da4c3f51fa93
I20240502 21:28:17.004537 28433 log_verifier.cc:126] Checking tablet be2cbcd79c1e40f49701271bb247f4d0
I20240502 21:28:17.357110 28433 log_verifier.cc:177] Verified matching terms for 956 ops in tablet be2cbcd79c1e40f49701271bb247f4d0
I20240502 21:28:17.358248 28433 log_verifier.cc:126] Checking tablet d4392d14cf254b72bf506f9c6cfdd55d
I20240502 21:28:17.686353 28433 log_verifier.cc:177] Verified matching terms for 860 ops in tablet d4392d14cf254b72bf506f9c6cfdd55d
I20240502 21:28:17.687444 28433 log_verifier.cc:126] Checking tablet d64bd5e7c22749078d82ad8da12d75dd
I20240502 21:28:18.047313 28433 log_verifier.cc:177] Verified matching terms for 946 ops in tablet d64bd5e7c22749078d82ad8da12d75dd
I20240502 21:28:18.048499 28433 log_verifier.cc:126] Checking tablet d7db9b73c0594885b383b059520414e9
I20240502 21:28:18.421964 28433 log_verifier.cc:177] Verified matching terms for 1000 ops in tablet d7db9b73c0594885b383b059520414e9
I20240502 21:28:18.423480 28433 log_verifier.cc:126] Checking tablet dc17a741596a4cb59692b7f13cf254e7
I20240502 21:28:18.852695 28433 log_verifier.cc:177] Verified matching terms for 1093 ops in tablet dc17a741596a4cb59692b7f13cf254e7
I20240502 21:28:18.854187 28433 log_verifier.cc:126] Checking tablet e93810d704674fc4ad4d96501fb5ea42
I20240502 21:28:19.199012 28433 log_verifier.cc:177] Verified matching terms for 901 ops in tablet e93810d704674fc4ad4d96501fb5ea42
I20240502 21:28:19.200322 28433 log_verifier.cc:126] Checking tablet fe578b007ccd496ba2f70cda8e90c91f
I20240502 21:28:19.619257 28433 log_verifier.cc:177] Verified matching terms for 968 ops in tablet fe578b007ccd496ba2f70cda8e90c91f
I20240502 21:28:19.636656 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.72:0
--local_ip_for_outbound_sockets=127.27.196.72
--webserver_interface=127.27.196.72
--webserver_port=0
--tserver_master_addrs=127.27.196.126:43629
--builtin_ntp_servers=127.27.196.84:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 21:28:20.136895  7618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:20.137586  7618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:20.137846  7618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:20.152208  7618 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:28:20.152443  7618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:20.152643  7618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.72
I20240502 21:28:20.164611  7618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.27.196.72
--webserver_port=0
--tserver_master_addrs=127.27.196.126:43629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.72
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:20.166594  7618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:20.169309  7618 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:20.190658  7625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:20.191462  7626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:20.191970  7628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:20.192658  7618 server_base.cc:1034] running on GCE node
I20240502 21:28:20.193470  7618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:20.194813  7618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:20.196189  7618 hybrid_clock.cc:648] HybridClock initialized: now 1714685300196009 us; error 134 us; skew 500 ppm
I20240502 21:28:20.196916  7618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:20.201869  7618 webserver.cc:460] Webserver started at http://127.27.196.72:44241/ using document root <none> and password file <none>
I20240502 21:28:20.203285  7618 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:20.203537  7618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:20.204041  7618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:20.209401  7618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "4ec1f2d4bde54666996929c96fa9a590"
format_stamp: "Formatted at 2024-05-02 21:28:20 on dist-test-slave-3bs1"
I20240502 21:28:20.210975  7618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "4ec1f2d4bde54666996929c96fa9a590"
format_stamp: "Formatted at 2024-05-02 21:28:20 on dist-test-slave-3bs1"
I20240502 21:28:20.221235  7618 fs_manager.cc:691] Time spent creating directory manager: real 0.010s	user 0.004s	sys 0.003s
I20240502 21:28:20.228523  7634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:28:20.231555  7618 fs_manager.cc:725] Time spent opening block manager: real 0.006s	user 0.006s	sys 0.000s
I20240502 21:28:20.231853  7618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "4ec1f2d4bde54666996929c96fa9a590"
format_stamp: "Formatted at 2024-05-02 21:28:20 on dist-test-slave-3bs1"
I20240502 21:28:20.232311  7618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:20.306088  7618 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:28:20.307633  7618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:20.308064  7618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:20.309880  7618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:20.312815  7618 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:20.312985  7618 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:28:20.313135  7618 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:20.313234  7618 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:28:20.372660  7618 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.72:44723
I20240502 21:28:20.372712  7750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.72:44723 every 8 connection(s)
I20240502 21:28:20.374792  7618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 21:28:20.385464 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 7618
I20240502 21:28:20.385718 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 21:28:20.394905  7751 heartbeater.cc:347] Connected to a master server at 127.27.196.126:43629
I20240502 21:28:20.395376  7751 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:20.396522  7751 heartbeater.cc:511] Master 127.27.196.126:43629 requested a full tablet report, sending...
I20240502 21:28:20.398519  2046 ts_manager.cc:194] Registered new tserver with Master: 4ec1f2d4bde54666996929c96fa9a590 (127.27.196.72:44723)
I20240502 21:28:20.399545  2046 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.72:34397
I20240502 21:28:21.059444  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:21.099041  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:21.121881  2308 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:21.134047  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:21.152691  3665 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:21.171434  7684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:21.174664  2174 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:21.188925  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:21.403384  7751 heartbeater.cc:503] Master 127.27.196.126:43629 was elected leader, sending a full tablet report...
I20240502 21:28:21.420552  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:21.425845  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:21.428107  2308 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:21.434499  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:21.442077  3665 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:21.446245  2174 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:21.448575  7684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:21.461097  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:21.627022  2329 consensus_queue.cc:237] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 922, Committed index: 922, Last appended: 1.922, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 923 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } }
I20240502 21:28:21.630205  2193 raft_consensus.cc:1261] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Refusing update from remote peer 68f426685b4c4c90bda61ef52177cc22: Log matching property violated. Preceding OpId in replica: term: 1 index: 922. Preceding OpId from leader: term: 1 index: 923. (index mismatch)
I20240502 21:28:21.631045  7619 consensus_queue.cc:1035] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 923, Last known committed idx: 922, Time since last communication: 0.000s
I20240502 21:28:21.634828  7619 raft_consensus.cc:2932] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Committing config change with OpId 1.923: config changed from index -1 to 923, VOTER 4ec1f2d4bde54666996929c96fa9a590 (127.27.196.72) added. New config: { opid_index: 923 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } } }
I20240502 21:28:21.635480  2193 raft_consensus.cc:2932] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Committing config change with OpId 1.923: config changed from index -1 to 923, VOTER 4ec1f2d4bde54666996929c96fa9a590 (127.27.196.72) added. New config: { opid_index: 923 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } } }
I20240502 21:28:21.640887  2047 catalog_manager.cc:5484] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 reported cstate change: config changed from index -1 to 923, VOTER 4ec1f2d4bde54666996929c96fa9a590 (127.27.196.72) added. New cstate: current_term: 1 leader_uuid: "68f426685b4c4c90bda61ef52177cc22" committed_config { opid_index: 923 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } } }
W20240502 21:28:21.662130  2262 consensus_peers.cc:485] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 -> Peer 4ec1f2d4bde54666996929c96fa9a590 (127.27.196.72:44723): Couldn't send request to peer 4ec1f2d4bde54666996929c96fa9a590. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a6da907ad0447b3ad2f66f50cd7289d. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:21.690011  2193 consensus_queue.cc:237] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1100, Committed index: 1100, Last appended: 1.1100, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1101 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } }
I20240502 21:28:21.693110  3816 raft_consensus.cc:1261] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Refusing update from remote peer d961985de7874e1990a19bb9b2da14c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1100. Preceding OpId from leader: term: 1 index: 1101. (index mismatch)
I20240502 21:28:21.693792  7884 consensus_queue.cc:1035] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1101, Last known committed idx: 1100, Time since last communication: 0.000s
I20240502 21:28:21.697789  7623 raft_consensus.cc:2932] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [term 1 LEADER]: Committing config change with OpId 1.1101: config changed from index 322 to 1101, VOTER 4ec1f2d4bde54666996929c96fa9a590 (127.27.196.72) added. New config: { opid_index: 1101 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } } }
I20240502 21:28:21.699669  3816 raft_consensus.cc:2932] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Committing config change with OpId 1.1101: config changed from index 322 to 1101, VOTER 4ec1f2d4bde54666996929c96fa9a590 (127.27.196.72) added. New config: { opid_index: 1101 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } } }
W20240502 21:28:21.700100  2127 consensus_peers.cc:485] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 -> Peer 4ec1f2d4bde54666996929c96fa9a590 (127.27.196.72:44723): Couldn't send request to peer 4ec1f2d4bde54666996929c96fa9a590. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 068c9a3114f648d1a183c9b90f4d365e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:21.706020  2046 catalog_manager.cc:5484] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 reported cstate change: config changed from index 322 to 1101, VOTER 4ec1f2d4bde54666996929c96fa9a590 (127.27.196.72) added. New cstate: current_term: 1 leader_uuid: "d961985de7874e1990a19bb9b2da14c2" committed_config { opid_index: 1101 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } } }
I20240502 21:28:21.750967  2466 consensus_queue.cc:237] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 860, Committed index: 860, Last appended: 1.860, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 861 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } }
I20240502 21:28:21.754042  3685 raft_consensus.cc:1261] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Refusing update from remote peer c616208720db4b14a9219cae8f2b7d49: Log matching property violated. Preceding OpId in replica: term: 1 index: 860. Preceding OpId from leader: term: 1 index: 861. (index mismatch)
I20240502 21:28:21.754699  7904 consensus_queue.cc:1035] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 861, Last known committed idx: 860, Time since last communication: 0.000s
I20240502 21:28:21.757598  7538 raft_consensus.cc:2932] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [term 1 LEADER]: Committing config change with OpId 1.861: config changed from index 312 to 861, VOTER 4ec1f2d4bde54666996929c96fa9a590 (127.27.196.72) added. New config: { opid_index: 861 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } } }
I20240502 21:28:21.758359  3685 raft_consensus.cc:2932] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Committing config change with OpId 1.861: config changed from index 312 to 861, VOTER 4ec1f2d4bde54666996929c96fa9a590 (127.27.196.72) added. New config: { opid_index: 861 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } } }
W20240502 21:28:21.760586  2398 consensus_peers.cc:485] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 -> Peer 4ec1f2d4bde54666996929c96fa9a590 (127.27.196.72:44723): Couldn't send request to peer 4ec1f2d4bde54666996929c96fa9a590. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4392d14cf254b72bf506f9c6cfdd55d. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:21.764955  2047 catalog_manager.cc:5484] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3 reported cstate change: config changed from index 312 to 861, VOTER 4ec1f2d4bde54666996929c96fa9a590 (127.27.196.72) added. New cstate: current_term: 1 leader_uuid: "c616208720db4b14a9219cae8f2b7d49" committed_config { opid_index: 861 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } } }
I20240502 21:28:21.808724  2466 consensus_queue.cc:237] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 968, Committed index: 968, Last appended: 1.968, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 969 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } }
I20240502 21:28:21.811547  3962 raft_consensus.cc:1261] T 0306f5bfea0a49eca06554a40ba50d34 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Refusing update from remote peer c616208720db4b14a9219cae8f2b7d49: Log matching property violated. Preceding OpId in replica: term: 1 index: 968. Preceding OpId from leader: term: 1 index: 969. (index mismatch)
W20240502 21:28:21.811761  2398 consensus_peers.cc:485] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 -> Peer 4ec1f2d4bde54666996929c96fa9a590 (127.27.196.72:44723): Couldn't send request to peer 4ec1f2d4bde54666996929c96fa9a590. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0306f5bfea0a49eca06554a40ba50d34. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:21.812166  7904 consensus_queue.cc:1035] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 969, Last known committed idx: 968, Time since last communication: 0.000s
I20240502 21:28:21.814971  7907 raft_consensus.cc:2932] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [term 1 LEADER]: Committing config change with OpId 1.969: config changed from index 251 to 969, VOTER 4ec1f2d4bde54666996929c96fa9a590 (127.27.196.72) added. New config: { opid_index: 969 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } } }
I20240502 21:28:21.815600  3962 raft_consensus.cc:2932] T 0306f5bfea0a49eca06554a40ba50d34 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Committing config change with OpId 1.969: config changed from index 251 to 969, VOTER 4ec1f2d4bde54666996929c96fa9a590 (127.27.196.72) added. New config: { opid_index: 969 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } } }
I20240502 21:28:21.821290  2047 catalog_manager.cc:5484] T 0306f5bfea0a49eca06554a40ba50d34 P d306b26127e04e8c85a98edf1b41978a reported cstate change: config changed from index 251 to 969, VOTER 4ec1f2d4bde54666996929c96fa9a590 (127.27.196.72) added. New cstate: current_term: 1 leader_uuid: "c616208720db4b14a9219cae8f2b7d49" committed_config { opid_index: 969 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } } }
I20240502 21:28:21.982717  7684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:21.983011  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:21.992175  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:21.996860  3665 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:22.011523  2308 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:22.028640  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:22.075512  2174 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:22.096300  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:22.137383  7971 ts_tablet_manager.cc:902] T 068c9a3114f648d1a183c9b90f4d365e P 4ec1f2d4bde54666996929c96fa9a590: Initiating tablet copy from peer d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025)
I20240502 21:28:22.150525  7971 tablet_copy_client.cc:308] T 068c9a3114f648d1a183c9b90f4d365e P 4ec1f2d4bde54666996929c96fa9a590: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:37025
I20240502 21:28:22.158635  2215 tablet_copy_service.cc:140] P d961985de7874e1990a19bb9b2da14c2: Received BeginTabletCopySession request for tablet 068c9a3114f648d1a183c9b90f4d365e from peer 4ec1f2d4bde54666996929c96fa9a590 ({username='slave'} at 127.27.196.72:50301)
I20240502 21:28:22.158833  2215 tablet_copy_service.cc:161] P d961985de7874e1990a19bb9b2da14c2: Beginning new tablet copy session on tablet 068c9a3114f648d1a183c9b90f4d365e from peer 4ec1f2d4bde54666996929c96fa9a590 at {username='slave'} at 127.27.196.72:50301: session id = 4ec1f2d4bde54666996929c96fa9a590-068c9a3114f648d1a183c9b90f4d365e
I20240502 21:28:22.162302  2215 tablet_copy_source_session.cc:205] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:22.170545  7971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 068c9a3114f648d1a183c9b90f4d365e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:22.187445  7971 tablet_copy_client.cc:783] T 068c9a3114f648d1a183c9b90f4d365e P 4ec1f2d4bde54666996929c96fa9a590: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:22.187893  7971 tablet_copy_client.cc:647] T 068c9a3114f648d1a183c9b90f4d365e P 4ec1f2d4bde54666996929c96fa9a590: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:22.196130  7971 tablet_copy_client.cc:523] T 068c9a3114f648d1a183c9b90f4d365e P 4ec1f2d4bde54666996929c96fa9a590: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:22.202558  7971 tablet_bootstrap.cc:492] T 068c9a3114f648d1a183c9b90f4d365e P 4ec1f2d4bde54666996929c96fa9a590: Bootstrap starting.
I20240502 21:28:22.203501  7983 ts_tablet_manager.cc:902] T 3a6da907ad0447b3ad2f66f50cd7289d P 4ec1f2d4bde54666996929c96fa9a590: Initiating tablet copy from peer 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253)
I20240502 21:28:22.205849  7983 tablet_copy_client.cc:308] T 3a6da907ad0447b3ad2f66f50cd7289d P 4ec1f2d4bde54666996929c96fa9a590: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:36253
I20240502 21:28:22.215626  2353 tablet_copy_service.cc:140] P 68f426685b4c4c90bda61ef52177cc22: Received BeginTabletCopySession request for tablet 3a6da907ad0447b3ad2f66f50cd7289d from peer 4ec1f2d4bde54666996929c96fa9a590 ({username='slave'} at 127.27.196.72:35559)
I20240502 21:28:22.215855  2353 tablet_copy_service.cc:161] P 68f426685b4c4c90bda61ef52177cc22: Beginning new tablet copy session on tablet 3a6da907ad0447b3ad2f66f50cd7289d from peer 4ec1f2d4bde54666996929c96fa9a590 at {username='slave'} at 127.27.196.72:35559: session id = 4ec1f2d4bde54666996929c96fa9a590-3a6da907ad0447b3ad2f66f50cd7289d
I20240502 21:28:22.219174  2353 tablet_copy_source_session.cc:205] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:22.221086  7983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a6da907ad0447b3ad2f66f50cd7289d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:22.228878  7983 tablet_copy_client.cc:783] T 3a6da907ad0447b3ad2f66f50cd7289d P 4ec1f2d4bde54666996929c96fa9a590: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:22.229238  7983 tablet_copy_client.cc:647] T 3a6da907ad0447b3ad2f66f50cd7289d P 4ec1f2d4bde54666996929c96fa9a590: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:22.234696  7983 tablet_copy_client.cc:523] T 3a6da907ad0447b3ad2f66f50cd7289d P 4ec1f2d4bde54666996929c96fa9a590: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:22.239331  7983 tablet_bootstrap.cc:492] T 3a6da907ad0447b3ad2f66f50cd7289d P 4ec1f2d4bde54666996929c96fa9a590: Bootstrap starting.
I20240502 21:28:22.396258  8005 ts_tablet_manager.cc:902] T d4392d14cf254b72bf506f9c6cfdd55d P 4ec1f2d4bde54666996929c96fa9a590: Initiating tablet copy from peer c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687)
I20240502 21:28:22.403031  8006 ts_tablet_manager.cc:902] T 0306f5bfea0a49eca06554a40ba50d34 P 4ec1f2d4bde54666996929c96fa9a590: Initiating tablet copy from peer c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687)
I20240502 21:28:22.419338  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:22.422387  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:22.432062  7684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:22.458325  3665 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:22.469169  8005 tablet_copy_client.cc:308] T d4392d14cf254b72bf506f9c6cfdd55d P 4ec1f2d4bde54666996929c96fa9a590: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:46687
I20240502 21:28:22.479420  2487 tablet_copy_service.cc:140] P c616208720db4b14a9219cae8f2b7d49: Received BeginTabletCopySession request for tablet d4392d14cf254b72bf506f9c6cfdd55d from peer 4ec1f2d4bde54666996929c96fa9a590 ({username='slave'} at 127.27.196.72:58627)
I20240502 21:28:22.479646  2487 tablet_copy_service.cc:161] P c616208720db4b14a9219cae8f2b7d49: Beginning new tablet copy session on tablet d4392d14cf254b72bf506f9c6cfdd55d from peer 4ec1f2d4bde54666996929c96fa9a590 at {username='slave'} at 127.27.196.72:58627: session id = 4ec1f2d4bde54666996929c96fa9a590-d4392d14cf254b72bf506f9c6cfdd55d
I20240502 21:28:22.483362  2487 tablet_copy_source_session.cc:205] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:22.485304  8005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4392d14cf254b72bf506f9c6cfdd55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:22.499990  8005 tablet_copy_client.cc:783] T d4392d14cf254b72bf506f9c6cfdd55d P 4ec1f2d4bde54666996929c96fa9a590: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:22.500763  8005 tablet_copy_client.cc:647] T d4392d14cf254b72bf506f9c6cfdd55d P 4ec1f2d4bde54666996929c96fa9a590: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:22.507529  2308 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:22.510833  8005 tablet_copy_client.cc:523] T d4392d14cf254b72bf506f9c6cfdd55d P 4ec1f2d4bde54666996929c96fa9a590: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:22.516000  8005 tablet_bootstrap.cc:492] T d4392d14cf254b72bf506f9c6cfdd55d P 4ec1f2d4bde54666996929c96fa9a590: Bootstrap starting.
I20240502 21:28:22.516198  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:22.534714  8006 tablet_copy_client.cc:308] T 0306f5bfea0a49eca06554a40ba50d34 P 4ec1f2d4bde54666996929c96fa9a590: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:46687
I20240502 21:28:22.535025  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:22.537312  2487 tablet_copy_service.cc:140] P c616208720db4b14a9219cae8f2b7d49: Received BeginTabletCopySession request for tablet 0306f5bfea0a49eca06554a40ba50d34 from peer 4ec1f2d4bde54666996929c96fa9a590 ({username='slave'} at 127.27.196.72:58627)
I20240502 21:28:22.537513  2487 tablet_copy_service.cc:161] P c616208720db4b14a9219cae8f2b7d49: Beginning new tablet copy session on tablet 0306f5bfea0a49eca06554a40ba50d34 from peer 4ec1f2d4bde54666996929c96fa9a590 at {username='slave'} at 127.27.196.72:58627: session id = 4ec1f2d4bde54666996929c96fa9a590-0306f5bfea0a49eca06554a40ba50d34
I20240502 21:28:22.540870  2487 tablet_copy_source_session.cc:205] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:22.547689  2174 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:22.548009  8006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0306f5bfea0a49eca06554a40ba50d34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:22.579989  7971 log.cc:824] T 068c9a3114f648d1a183c9b90f4d365e P 4ec1f2d4bde54666996929c96fa9a590: Log is configured to *not* fsync() on all Append() calls
I20240502 21:28:22.630759  8006 tablet_copy_client.cc:783] T 0306f5bfea0a49eca06554a40ba50d34 P 4ec1f2d4bde54666996929c96fa9a590: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:22.631963  8006 tablet_copy_client.cc:647] T 0306f5bfea0a49eca06554a40ba50d34 P 4ec1f2d4bde54666996929c96fa9a590: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:22.649379  8006 tablet_copy_client.cc:523] T 0306f5bfea0a49eca06554a40ba50d34 P 4ec1f2d4bde54666996929c96fa9a590: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:22.653728  8006 tablet_bootstrap.cc:492] T 0306f5bfea0a49eca06554a40ba50d34 P 4ec1f2d4bde54666996929c96fa9a590: Bootstrap starting.
I20240502 21:28:23.332149  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:23.350178  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:23.391952  3665 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:23.414253  7684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:23.422955  2308 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:23.475059  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:23.507139  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:23.511557  2174 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:28:23.615283  7971 log.cc:924] Time spent T 068c9a3114f648d1a183c9b90f4d365e P 4ec1f2d4bde54666996929c96fa9a590: Append to log took a long time: real 0.092s	user 0.004s	sys 0.000s
I20240502 21:28:24.071602  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:24.148576  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:24.163614  3665 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:24.221702  7684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:24.246088  2308 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:24.270264  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:24.296895  2174 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:24.311684  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 21:28:24.367636  2262 consensus_peers.cc:485] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 -> Peer 4ec1f2d4bde54666996929c96fa9a590 (127.27.196.72:44723): Couldn't send request to peer 4ec1f2d4bde54666996929c96fa9a590. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240502 21:28:24.679371  2127 consensus_peers.cc:485] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 -> Peer 4ec1f2d4bde54666996929c96fa9a590 (127.27.196.72:44723): Couldn't send request to peer 4ec1f2d4bde54666996929c96fa9a590. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240502 21:28:24.866704  7983 tablet_bootstrap.cc:492] T 3a6da907ad0447b3ad2f66f50cd7289d P 4ec1f2d4bde54666996929c96fa9a590: Bootstrap replayed 1/1 log segments. Stats: ops{read=923 overwritten=0 applied=923 ignored=0} inserts{seen=1595 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:24.867508  7983 tablet_bootstrap.cc:492] T 3a6da907ad0447b3ad2f66f50cd7289d P 4ec1f2d4bde54666996929c96fa9a590: Bootstrap complete.
I20240502 21:28:24.868305  7983 ts_tablet_manager.cc:1367] T 3a6da907ad0447b3ad2f66f50cd7289d P 4ec1f2d4bde54666996929c96fa9a590: Time spent bootstrapping tablet: real 2.629s	user 1.191s	sys 0.171s
I20240502 21:28:24.893651  7983 raft_consensus.cc:348] T 3a6da907ad0447b3ad2f66f50cd7289d P 4ec1f2d4bde54666996929c96fa9a590 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 923 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } }
I20240502 21:28:24.894449  7983 raft_consensus.cc:729] T 3a6da907ad0447b3ad2f66f50cd7289d P 4ec1f2d4bde54666996929c96fa9a590 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ec1f2d4bde54666996929c96fa9a590, State: Initialized, Role: FOLLOWER
I20240502 21:28:24.895501  7983 consensus_queue.cc:260] T 3a6da907ad0447b3ad2f66f50cd7289d P 4ec1f2d4bde54666996929c96fa9a590 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 923, Last appended: 1.923, Last appended by leader: 923, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 923 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } }
I20240502 21:28:24.908365  7983 ts_tablet_manager.cc:1398] T 3a6da907ad0447b3ad2f66f50cd7289d P 4ec1f2d4bde54666996929c96fa9a590: Time spent starting tablet: real 0.039s	user 0.010s	sys 0.005s
I20240502 21:28:24.909677  2353 tablet_copy_service.cc:342] P 68f426685b4c4c90bda61ef52177cc22: Request end of tablet copy session 4ec1f2d4bde54666996929c96fa9a590-3a6da907ad0447b3ad2f66f50cd7289d received from {username='slave'} at 127.27.196.72:35559
I20240502 21:28:24.909865  2353 tablet_copy_service.cc:433] P 68f426685b4c4c90bda61ef52177cc22: ending tablet copy session 4ec1f2d4bde54666996929c96fa9a590-3a6da907ad0447b3ad2f66f50cd7289d on tablet 3a6da907ad0447b3ad2f66f50cd7289d with peer 4ec1f2d4bde54666996929c96fa9a590
W20240502 21:28:24.926681  2398 consensus_peers.cc:485] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 -> Peer 4ec1f2d4bde54666996929c96fa9a590 (127.27.196.72:44723): Couldn't send request to peer 4ec1f2d4bde54666996929c96fa9a590. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240502 21:28:24.936240  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:24.964747  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:24.974967  3665 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:25.018791  7684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:25.050104  2174 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:25.059998  8005 tablet_bootstrap.cc:492] T d4392d14cf254b72bf506f9c6cfdd55d P 4ec1f2d4bde54666996929c96fa9a590: Bootstrap replayed 1/1 log segments. Stats: ops{read=861 overwritten=0 applied=861 ignored=0} inserts{seen=1544 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:25.060604  8005 tablet_bootstrap.cc:492] T d4392d14cf254b72bf506f9c6cfdd55d P 4ec1f2d4bde54666996929c96fa9a590: Bootstrap complete.
I20240502 21:28:25.061064  8005 ts_tablet_manager.cc:1367] T d4392d14cf254b72bf506f9c6cfdd55d P 4ec1f2d4bde54666996929c96fa9a590: Time spent bootstrapping tablet: real 2.545s	user 1.091s	sys 0.158s
I20240502 21:28:25.062398  8005 raft_consensus.cc:348] T d4392d14cf254b72bf506f9c6cfdd55d P 4ec1f2d4bde54666996929c96fa9a590 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 861 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } }
I20240502 21:28:25.062785  8005 raft_consensus.cc:729] T d4392d14cf254b72bf506f9c6cfdd55d P 4ec1f2d4bde54666996929c96fa9a590 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ec1f2d4bde54666996929c96fa9a590, State: Initialized, Role: FOLLOWER
I20240502 21:28:25.063181  8005 consensus_queue.cc:260] T d4392d14cf254b72bf506f9c6cfdd55d P 4ec1f2d4bde54666996929c96fa9a590 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 861, Last appended: 1.861, Last appended by leader: 861, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 861 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } }
I20240502 21:28:25.063932  8005 ts_tablet_manager.cc:1398] T d4392d14cf254b72bf506f9c6cfdd55d P 4ec1f2d4bde54666996929c96fa9a590: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 21:28:25.064354  2308 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:25.138249  7703 raft_consensus.cc:1203] T d4392d14cf254b72bf506f9c6cfdd55d P 4ec1f2d4bde54666996929c96fa9a590 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.860->[1.861-1.861]   Dedup: 1.861->[]
I20240502 21:28:25.151445  2487 tablet_copy_service.cc:342] P c616208720db4b14a9219cae8f2b7d49: Request end of tablet copy session 4ec1f2d4bde54666996929c96fa9a590-d4392d14cf254b72bf506f9c6cfdd55d received from {username='slave'} at 127.27.196.72:58627
I20240502 21:28:25.151618  2487 tablet_copy_service.cc:433] P c616208720db4b14a9219cae8f2b7d49: ending tablet copy session 4ec1f2d4bde54666996929c96fa9a590-d4392d14cf254b72bf506f9c6cfdd55d on tablet d4392d14cf254b72bf506f9c6cfdd55d with peer 4ec1f2d4bde54666996929c96fa9a590
I20240502 21:28:25.158064  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:25.167889  7971 tablet_bootstrap.cc:492] T 068c9a3114f648d1a183c9b90f4d365e P 4ec1f2d4bde54666996929c96fa9a590: Bootstrap replayed 1/1 log segments. Stats: ops{read=1101 overwritten=0 applied=1101 ignored=0} inserts{seen=1771 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:25.168596  7971 tablet_bootstrap.cc:492] T 068c9a3114f648d1a183c9b90f4d365e P 4ec1f2d4bde54666996929c96fa9a590: Bootstrap complete.
I20240502 21:28:25.169109  7971 ts_tablet_manager.cc:1367] T 068c9a3114f648d1a183c9b90f4d365e P 4ec1f2d4bde54666996929c96fa9a590: Time spent bootstrapping tablet: real 2.967s	user 1.439s	sys 0.133s
I20240502 21:28:25.170065  7971 raft_consensus.cc:348] T 068c9a3114f648d1a183c9b90f4d365e P 4ec1f2d4bde54666996929c96fa9a590 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1101 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } }
I20240502 21:28:25.170408  7971 raft_consensus.cc:729] T 068c9a3114f648d1a183c9b90f4d365e P 4ec1f2d4bde54666996929c96fa9a590 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ec1f2d4bde54666996929c96fa9a590, State: Initialized, Role: FOLLOWER
I20240502 21:28:25.170836  7971 consensus_queue.cc:260] T 068c9a3114f648d1a183c9b90f4d365e P 4ec1f2d4bde54666996929c96fa9a590 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1101, Last appended: 1.1101, Last appended by leader: 1101, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1101 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } }
I20240502 21:28:25.171274  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:25.171736  7971 ts_tablet_manager.cc:1398] T 068c9a3114f648d1a183c9b90f4d365e P 4ec1f2d4bde54666996929c96fa9a590: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:28:25.172964  2215 tablet_copy_service.cc:342] P d961985de7874e1990a19bb9b2da14c2: Request end of tablet copy session 4ec1f2d4bde54666996929c96fa9a590-068c9a3114f648d1a183c9b90f4d365e received from {username='slave'} at 127.27.196.72:50301
I20240502 21:28:25.173184  2215 tablet_copy_service.cc:433] P d961985de7874e1990a19bb9b2da14c2: ending tablet copy session 4ec1f2d4bde54666996929c96fa9a590-068c9a3114f648d1a183c9b90f4d365e on tablet 068c9a3114f648d1a183c9b90f4d365e with peer 4ec1f2d4bde54666996929c96fa9a590
I20240502 21:28:25.213485  8006 tablet_bootstrap.cc:492] T 0306f5bfea0a49eca06554a40ba50d34 P 4ec1f2d4bde54666996929c96fa9a590: Bootstrap replayed 1/1 log segments. Stats: ops{read=969 overwritten=0 applied=969 ignored=0} inserts{seen=1652 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:25.214129  8006 tablet_bootstrap.cc:492] T 0306f5bfea0a49eca06554a40ba50d34 P 4ec1f2d4bde54666996929c96fa9a590: Bootstrap complete.
I20240502 21:28:25.214623  8006 ts_tablet_manager.cc:1367] T 0306f5bfea0a49eca06554a40ba50d34 P 4ec1f2d4bde54666996929c96fa9a590: Time spent bootstrapping tablet: real 2.561s	user 1.241s	sys 0.102s
I20240502 21:28:25.215667  8006 raft_consensus.cc:348] T 0306f5bfea0a49eca06554a40ba50d34 P 4ec1f2d4bde54666996929c96fa9a590 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 969 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } }
I20240502 21:28:25.215965  8006 raft_consensus.cc:729] T 0306f5bfea0a49eca06554a40ba50d34 P 4ec1f2d4bde54666996929c96fa9a590 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ec1f2d4bde54666996929c96fa9a590, State: Initialized, Role: FOLLOWER
I20240502 21:28:25.216315  8006 consensus_queue.cc:260] T 0306f5bfea0a49eca06554a40ba50d34 P 4ec1f2d4bde54666996929c96fa9a590 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 969, Last appended: 1.969, Last appended by leader: 969, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 969 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } }
I20240502 21:28:25.217734  8006 ts_tablet_manager.cc:1398] T 0306f5bfea0a49eca06554a40ba50d34 P 4ec1f2d4bde54666996929c96fa9a590: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 21:28:25.219913  2487 tablet_copy_service.cc:342] P c616208720db4b14a9219cae8f2b7d49: Request end of tablet copy session 4ec1f2d4bde54666996929c96fa9a590-0306f5bfea0a49eca06554a40ba50d34 received from {username='slave'} at 127.27.196.72:58627
I20240502 21:28:25.220068  2487 tablet_copy_service.cc:433] P c616208720db4b14a9219cae8f2b7d49: ending tablet copy session 4ec1f2d4bde54666996929c96fa9a590-0306f5bfea0a49eca06554a40ba50d34 on tablet 0306f5bfea0a49eca06554a40ba50d34 with peer 4ec1f2d4bde54666996929c96fa9a590
I20240502 21:28:25.390724  7703 raft_consensus.cc:1203] T 0306f5bfea0a49eca06554a40ba50d34 P 4ec1f2d4bde54666996929c96fa9a590 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.968->[1.969-1.969]   Dedup: 1.969->[]
I20240502 21:28:25.423693  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:25.425189  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:25.470489  7684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:25.477382  3665 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:25.478840  2308 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:25.480432  7703 raft_consensus.cc:1203] T 3a6da907ad0447b3ad2f66f50cd7289d P 4ec1f2d4bde54666996929c96fa9a590 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.922->[1.923-1.923]   Dedup: 1.923->[]
I20240502 21:28:25.515564  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:25.517741  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 21:28:25.542977  2130 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.68:44575, user_credentials={real_user=slave}} blocked reactor thread for 47450.3us
I20240502 21:28:25.555184  2174 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:25.563634  7703 raft_consensus.cc:1203] T 068c9a3114f648d1a183c9b90f4d365e P 4ec1f2d4bde54666996929c96fa9a590 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1100->[1.1101-1.1101]   Dedup: 1.1101->[]
I20240502 21:28:25.849062  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:25.850503  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:25.866962  7684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:25.871448  3665 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:25.879031  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:25.891744  2308 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:25.893693  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:25.902240  2174 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:26.341393  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:26.341533  7684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:26.345638  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:26.357496  2447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:26.358068  2308 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:26.380209  2174 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:26.393431  3665 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:26.398676  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:26.526293  2466 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d4392d14cf254b72bf506f9c6cfdd55d"
dest_uuid: "c616208720db4b14a9219cae8f2b7d49"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:53932
I20240502 21:28:26.526654  2466 raft_consensus.cc:595] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:28:26.661872  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:26.664633  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:26.671541  8273 raft_consensus.cc:979] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49: : Instructing follower 4e8c16d1a1484a41a54b5ddc198becd3 to start an election
I20240502 21:28:26.671783  8273 raft_consensus.cc:1067] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [term 1 LEADER]: Signalling peer 4e8c16d1a1484a41a54b5ddc198becd3 to start an election
I20240502 21:28:26.674126  3685 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d4392d14cf254b72bf506f9c6cfdd55d"
dest_uuid: "4e8c16d1a1484a41a54b5ddc198becd3"
 from {username='slave'} at 127.27.196.67:56237
I20240502 21:28:26.674427  3685 raft_consensus.cc:482] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:28:26.674556  3685 raft_consensus.cc:3037] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:26.677712  3685 raft_consensus.cc:504] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 861 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } }
I20240502 21:28:26.679719  2466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4392d14cf254b72bf506f9c6cfdd55d" candidate_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 861 } } ignore_live_leader: true dest_uuid: "c616208720db4b14a9219cae8f2b7d49"
I20240502 21:28:26.679980  2466 raft_consensus.cc:3032] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:28:26.680089  2466 raft_consensus.cc:729] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c616208720db4b14a9219cae8f2b7d49, State: Running, Role: LEADER
I20240502 21:28:26.680588  2466 consensus_queue.cc:260] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 861, Committed index: 861, Last appended: 1.861, Last appended by leader: 861, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 861 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } }
I20240502 21:28:26.680999  2466 raft_consensus.cc:3037] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:26.683074  2466 raft_consensus.cc:2454] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e8c16d1a1484a41a54b5ddc198becd3 in term 2.
I20240502 21:28:26.694219  3619 leader_election.cc:302] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e8c16d1a1484a41a54b5ddc198becd3, c616208720db4b14a9219cae8f2b7d49; no voters: 
I20240502 21:28:26.694691  7611 raft_consensus.cc:2781] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:28:26.700853  3685 leader_election.cc:288] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3 [CANDIDATE]: Term 2 election: Requested vote from peers c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687), 4ec1f2d4bde54666996929c96fa9a590 (127.27.196.72:44723)
I20240502 21:28:26.706482  2308 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:26.707051  7684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:26.720397  7703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4392d14cf254b72bf506f9c6cfdd55d" candidate_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 861 } } ignore_live_leader: true dest_uuid: "4ec1f2d4bde54666996929c96fa9a590"
I20240502 21:28:26.721045  7703 raft_consensus.cc:3037] T d4392d14cf254b72bf506f9c6cfdd55d P 4ec1f2d4bde54666996929c96fa9a590 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:26.723676  7611 raft_consensus.cc:686] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3 [term 2 LEADER]: Becoming Leader. State: Replica: 4e8c16d1a1484a41a54b5ddc198becd3, State: Running, Role: LEADER
I20240502 21:28:26.724112  7611 consensus_queue.cc:237] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 861, Committed index: 861, Last appended: 1.861, Last appended by leader: 861, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 861 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } }
I20240502 21:28:26.731698  7703 raft_consensus.cc:2454] T d4392d14cf254b72bf506f9c6cfdd55d P 4ec1f2d4bde54666996929c96fa9a590 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e8c16d1a1484a41a54b5ddc198becd3 in term 2.
I20240502 21:28:26.731820  2050 catalog_manager.cc:5484] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3 reported cstate change: term changed from 1 to 2, leader changed from c616208720db4b14a9219cae8f2b7d49 (127.27.196.67) to 4e8c16d1a1484a41a54b5ddc198becd3 (127.27.196.69). New cstate: current_term: 2 leader_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" committed_config { opid_index: 861 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } } }
I20240502 21:28:26.739790  3665 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:26.760715  2447 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:26.764088  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:26.785149  2174 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:27.018630  2308 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:27.019948  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:27.022795  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:27.024188  7684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:27.034885  2447 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:27.042945  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:27.070899  2174 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:27.073029  3665 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:27.118350  2466 raft_consensus.cc:1261] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [term 2 FOLLOWER]: Refusing update from remote peer 4e8c16d1a1484a41a54b5ddc198becd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 861. Preceding OpId from leader: term: 2 index: 862. (index mismatch)
I20240502 21:28:27.119192  7611 consensus_queue.cc:1035] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 862, Last known committed idx: 861, Time since last communication: 0.000s
I20240502 21:28:27.171595  7703 raft_consensus.cc:1261] T d4392d14cf254b72bf506f9c6cfdd55d P 4ec1f2d4bde54666996929c96fa9a590 [term 2 FOLLOWER]: Refusing update from remote peer 4e8c16d1a1484a41a54b5ddc198becd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 861. Preceding OpId from leader: term: 2 index: 862. (index mismatch)
I20240502 21:28:27.176769  7611 consensus_queue.cc:1035] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 862, Last known committed idx: 861, Time since last communication: 0.001s
I20240502 21:28:27.204283  2329 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3a6da907ad0447b3ad2f66f50cd7289d"
dest_uuid: "68f426685b4c4c90bda61ef52177cc22"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:58284
I20240502 21:28:27.204526  2329 raft_consensus.cc:595] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:28:27.237563  8156 raft_consensus.cc:979] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22: : Instructing follower d961985de7874e1990a19bb9b2da14c2 to start an election
I20240502 21:28:27.237855  8156 raft_consensus.cc:1067] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Signalling peer d961985de7874e1990a19bb9b2da14c2 to start an election
I20240502 21:28:27.239032  2193 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3a6da907ad0447b3ad2f66f50cd7289d"
dest_uuid: "d961985de7874e1990a19bb9b2da14c2"
 from {username='slave'} at 127.27.196.66:33861
I20240502 21:28:27.239437  2193 raft_consensus.cc:482] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:28:27.239563  2193 raft_consensus.cc:3037] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:27.242993  2193 raft_consensus.cc:504] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 923 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } }
I20240502 21:28:27.244107  2193 leader_election.cc:288] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2 [CANDIDATE]: Term 2 election: Requested vote from peers 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253), 4ec1f2d4bde54666996929c96fa9a590 (127.27.196.72:44723)
I20240502 21:28:27.247203  7703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a6da907ad0447b3ad2f66f50cd7289d" candidate_uuid: "d961985de7874e1990a19bb9b2da14c2" candidate_term: 2 candidate_status { last_received { term: 1 index: 923 } } ignore_live_leader: true dest_uuid: "4ec1f2d4bde54666996929c96fa9a590"
I20240502 21:28:27.247471  7703 raft_consensus.cc:3037] T 3a6da907ad0447b3ad2f66f50cd7289d P 4ec1f2d4bde54666996929c96fa9a590 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:27.250216  7703 raft_consensus.cc:2454] T 3a6da907ad0447b3ad2f66f50cd7289d P 4ec1f2d4bde54666996929c96fa9a590 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d961985de7874e1990a19bb9b2da14c2 in term 2.
I20240502 21:28:27.250824  2127 leader_election.cc:302] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ec1f2d4bde54666996929c96fa9a590, d961985de7874e1990a19bb9b2da14c2; no voters: 
I20240502 21:28:27.251263  7623 raft_consensus.cc:2781] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:28:27.251809  7623 raft_consensus.cc:686] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2 [term 2 LEADER]: Becoming Leader. State: Replica: d961985de7874e1990a19bb9b2da14c2, State: Running, Role: LEADER
I20240502 21:28:27.252182  7623 consensus_queue.cc:237] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 923, Committed index: 923, Last appended: 1.923, Last appended by leader: 923, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 923 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } }
I20240502 21:28:27.256170  2043 catalog_manager.cc:5484] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2 reported cstate change: term changed from 1 to 2, leader changed from 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66) to d961985de7874e1990a19bb9b2da14c2 (127.27.196.65). New cstate: current_term: 2 leader_uuid: "d961985de7874e1990a19bb9b2da14c2" committed_config { opid_index: 923 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } } }
I20240502 21:28:27.256919  2329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a6da907ad0447b3ad2f66f50cd7289d" candidate_uuid: "d961985de7874e1990a19bb9b2da14c2" candidate_term: 2 candidate_status { last_received { term: 1 index: 923 } } ignore_live_leader: true dest_uuid: "68f426685b4c4c90bda61ef52177cc22"
I20240502 21:28:27.257352  2329 raft_consensus.cc:3032] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:28:27.257469  2329 raft_consensus.cc:729] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 68f426685b4c4c90bda61ef52177cc22, State: Running, Role: LEADER
I20240502 21:28:27.257908  2329 consensus_queue.cc:260] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 923, Committed index: 923, Last appended: 1.923, Last appended by leader: 923, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 923 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } }
I20240502 21:28:27.258565  2329 raft_consensus.cc:3037] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:27.265267  2329 raft_consensus.cc:2454] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d961985de7874e1990a19bb9b2da14c2 in term 2.
I20240502 21:28:27.348613  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:27.350778  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:27.355803  7684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:27.364500  2447 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:27.391860  3665 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:27.413116  2174 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:27.414006  2308 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:27.435820  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:27.663031  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:27.665549  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:27.667143  3665 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:27.669148  2329 raft_consensus.cc:1261] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [term 2 FOLLOWER]: Refusing update from remote peer d961985de7874e1990a19bb9b2da14c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 923. Preceding OpId from leader: term: 2 index: 924. (index mismatch)
I20240502 21:28:27.670188  7623 consensus_queue.cc:1035] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 924, Last known committed idx: 923, Time since last communication: 0.000s
I20240502 21:28:27.691903  7684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:27.694054  7704 raft_consensus.cc:1261] T 3a6da907ad0447b3ad2f66f50cd7289d P 4ec1f2d4bde54666996929c96fa9a590 [term 2 FOLLOWER]: Refusing update from remote peer d961985de7874e1990a19bb9b2da14c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 923. Preceding OpId from leader: term: 2 index: 924. (index mismatch)
I20240502 21:28:27.696588  7623 consensus_queue.cc:1035] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 924, Last known committed idx: 923, Time since last communication: 0.000s
I20240502 21:28:27.703828  2447 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:27.736389  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:27.772572  2174 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:27.789652  2308 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:27.914103  2466 consensus_queue.cc:237] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 969, Committed index: 969, Last appended: 1.969, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 970 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } }
I20240502 21:28:27.917407  7704 raft_consensus.cc:1261] T 0306f5bfea0a49eca06554a40ba50d34 P 4ec1f2d4bde54666996929c96fa9a590 [term 1 FOLLOWER]: Refusing update from remote peer c616208720db4b14a9219cae8f2b7d49: Log matching property violated. Preceding OpId in replica: term: 1 index: 969. Preceding OpId from leader: term: 1 index: 970. (index mismatch)
I20240502 21:28:27.918264  8004 consensus_queue.cc:1035] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 970, Last known committed idx: 969, Time since last communication: 0.000s
I20240502 21:28:27.922268  8004 raft_consensus.cc:2932] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 [term 1 LEADER]: Committing config change with OpId 1.970: config changed from index 969 to 970, VOTER d306b26127e04e8c85a98edf1b41978a (127.27.196.71) evicted. New config: { opid_index: 970 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } } }
I20240502 21:28:27.925876  7704 raft_consensus.cc:2932] T 0306f5bfea0a49eca06554a40ba50d34 P 4ec1f2d4bde54666996929c96fa9a590 [term 1 FOLLOWER]: Committing config change with OpId 1.970: config changed from index 969 to 970, VOTER d306b26127e04e8c85a98edf1b41978a (127.27.196.71) evicted. New config: { opid_index: 970 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } } }
I20240502 21:28:27.933183  2050 catalog_manager.cc:5484] T 0306f5bfea0a49eca06554a40ba50d34 P c616208720db4b14a9219cae8f2b7d49 reported cstate change: config changed from index 969 to 970, VOTER d306b26127e04e8c85a98edf1b41978a (127.27.196.71) evicted. New cstate: current_term: 1 leader_uuid: "c616208720db4b14a9219cae8f2b7d49" committed_config { opid_index: 970 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } } }
I20240502 21:28:27.938477  3944 tablet_service.cc:1514] Processing DeleteTablet for tablet 0306f5bfea0a49eca06554a40ba50d34 with delete_type TABLET_DATA_TOMBSTONED (TS d306b26127e04e8c85a98edf1b41978a not found in new config with opid_index 970) from {username='slave'} at 127.0.0.1:42656
I20240502 21:28:27.939282  8478 tablet_replica.cc:330] T 0306f5bfea0a49eca06554a40ba50d34 P d306b26127e04e8c85a98edf1b41978a: stopping tablet replica
I20240502 21:28:27.939514  8478 raft_consensus.cc:2229] T 0306f5bfea0a49eca06554a40ba50d34 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:27.939819  8478 raft_consensus.cc:2258] T 0306f5bfea0a49eca06554a40ba50d34 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:27.941802  8478 ts_tablet_manager.cc:1875] T 0306f5bfea0a49eca06554a40ba50d34 P d306b26127e04e8c85a98edf1b41978a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:27.947615  8478 ts_tablet_manager.cc:1888] T 0306f5bfea0a49eca06554a40ba50d34 P d306b26127e04e8c85a98edf1b41978a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.969
I20240502 21:28:27.947844  8478 log.cc:1196] T 0306f5bfea0a49eca06554a40ba50d34 P d306b26127e04e8c85a98edf1b41978a: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal/wals/0306f5bfea0a49eca06554a40ba50d34
I20240502 21:28:27.948787  2036 catalog_manager.cc:4830] TS d306b26127e04e8c85a98edf1b41978a (127.27.196.71:39583): tablet 0306f5bfea0a49eca06554a40ba50d34 (table rebalance_test_table_4 [id=69e9b99652e74772bfdd02eeaf9e1bd1]) successfully deleted
I20240502 21:28:28.074919  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:28.083659  7684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:28.085395  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:28.099061  2308 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:28.108713  3665 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:28.137876  2447 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:28.182267  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:28.195188  2174 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:28.253733  2193 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "068c9a3114f648d1a183c9b90f4d365e"
dest_uuid: "d961985de7874e1990a19bb9b2da14c2"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34516
I20240502 21:28:28.253968  2193 raft_consensus.cc:595] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:28:28.371141  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:28.408901  2308 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:28.410332  3665 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:28.410425  7684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:28.433022  2447 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:28.456995  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:28.472388  3798 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:28.510236  2174 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:28.704444  8378 raft_consensus.cc:979] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2: : Instructing follower 1193f1d0d2c843139d98458cf4035eb8 to start an election
I20240502 21:28:28.705309  8552 raft_consensus.cc:1067] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [term 1 LEADER]: Signalling peer 1193f1d0d2c843139d98458cf4035eb8 to start an election
I20240502 21:28:28.706439  3814 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "068c9a3114f648d1a183c9b90f4d365e"
dest_uuid: "1193f1d0d2c843139d98458cf4035eb8"
 from {username='slave'} at 127.27.196.65:47645
I20240502 21:28:28.706892  3814 raft_consensus.cc:482] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:28:28.707089  3814 raft_consensus.cc:3037] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:28.710379  3814 raft_consensus.cc:504] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 1101 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } }
I20240502 21:28:28.712572  3814 leader_election.cc:288] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8 [CANDIDATE]: Term 2 election: Requested vote from peers d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025), 4ec1f2d4bde54666996929c96fa9a590 (127.27.196.72:44723)
I20240502 21:28:28.720986  2193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "068c9a3114f648d1a183c9b90f4d365e" candidate_uuid: "1193f1d0d2c843139d98458cf4035eb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1101 } } ignore_live_leader: true dest_uuid: "d961985de7874e1990a19bb9b2da14c2"
I20240502 21:28:28.721321  2193 raft_consensus.cc:3032] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:28:28.721181  7704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "068c9a3114f648d1a183c9b90f4d365e" candidate_uuid: "1193f1d0d2c843139d98458cf4035eb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1101 } } ignore_live_leader: true dest_uuid: "4ec1f2d4bde54666996929c96fa9a590"
I20240502 21:28:28.721518  2193 raft_consensus.cc:729] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d961985de7874e1990a19bb9b2da14c2, State: Running, Role: LEADER
I20240502 21:28:28.721601  7704 raft_consensus.cc:3037] T 068c9a3114f648d1a183c9b90f4d365e P 4ec1f2d4bde54666996929c96fa9a590 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:28.721851  2193 consensus_queue.cc:260] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1101, Committed index: 1101, Last appended: 1.1101, Last appended by leader: 1101, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1101 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } }
I20240502 21:28:28.722282  2193 raft_consensus.cc:3037] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:28.724058  7704 raft_consensus.cc:2454] T 068c9a3114f648d1a183c9b90f4d365e P 4ec1f2d4bde54666996929c96fa9a590 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1193f1d0d2c843139d98458cf4035eb8 in term 2.
I20240502 21:28:28.724859  2193 raft_consensus.cc:2454] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1193f1d0d2c843139d98458cf4035eb8 in term 2.
I20240502 21:28:28.724834  3751 leader_election.cc:302] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1193f1d0d2c843139d98458cf4035eb8, 4ec1f2d4bde54666996929c96fa9a590; no voters: 
I20240502 21:28:28.725577  8556 raft_consensus.cc:2781] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:28:28.726022  8556 raft_consensus.cc:686] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8 [term 2 LEADER]: Becoming Leader. State: Replica: 1193f1d0d2c843139d98458cf4035eb8, State: Running, Role: LEADER
I20240502 21:28:28.726495  8556 consensus_queue.cc:237] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1101, Committed index: 1101, Last appended: 1.1101, Last appended by leader: 1101, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1101 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } }
I20240502 21:28:28.730057  2050 catalog_manager.cc:5484] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8 reported cstate change: term changed from 1 to 2, leader changed from d961985de7874e1990a19bb9b2da14c2 (127.27.196.65) to 1193f1d0d2c843139d98458cf4035eb8 (127.27.196.70). New cstate: current_term: 2 leader_uuid: "1193f1d0d2c843139d98458cf4035eb8" committed_config { opid_index: 1101 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } } }
I20240502 21:28:28.943253  3798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:28.950737  3665 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:28.965406  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:28.986181  2308 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:29.001493  2174 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:29.003175  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:29.007491  2447 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:29.020699  7684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:29.108079  7704 raft_consensus.cc:1261] T 068c9a3114f648d1a183c9b90f4d365e P 4ec1f2d4bde54666996929c96fa9a590 [term 2 FOLLOWER]: Refusing update from remote peer 1193f1d0d2c843139d98458cf4035eb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1101. Preceding OpId from leader: term: 2 index: 1102. (index mismatch)
I20240502 21:28:29.109170  8556 consensus_queue.cc:1035] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1102, Last known committed idx: 1101, Time since last communication: 0.000s
I20240502 21:28:29.119633  2193 raft_consensus.cc:1261] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [term 2 FOLLOWER]: Refusing update from remote peer 1193f1d0d2c843139d98458cf4035eb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1101. Preceding OpId from leader: term: 2 index: 1102. (index mismatch)
I20240502 21:28:29.121215  8557 consensus_queue.cc:1035] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1102, Last known committed idx: 1101, Time since last communication: 0.000s
I20240502 21:28:29.164218  3685 consensus_queue.cc:237] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 862, Committed index: 862, Last appended: 2.862, Last appended by leader: 861, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 863 OBSOLETE_local: false peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } }
I20240502 21:28:29.166374  7704 raft_consensus.cc:1261] T d4392d14cf254b72bf506f9c6cfdd55d P 4ec1f2d4bde54666996929c96fa9a590 [term 2 FOLLOWER]: Refusing update from remote peer 4e8c16d1a1484a41a54b5ddc198becd3: Log matching property violated. Preceding OpId in replica: term: 2 index: 862. Preceding OpId from leader: term: 2 index: 863. (index mismatch)
I20240502 21:28:29.167076  7611 consensus_queue.cc:1035] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 863, Last known committed idx: 862, Time since last communication: 0.000s
I20240502 21:28:29.171161  8627 raft_consensus.cc:2932] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3 [term 2 LEADER]: Committing config change with OpId 2.863: config changed from index 861 to 863, VOTER c616208720db4b14a9219cae8f2b7d49 (127.27.196.67) evicted. New config: { opid_index: 863 OBSOLETE_local: false peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } } }
I20240502 21:28:29.172358  7704 raft_consensus.cc:2932] T d4392d14cf254b72bf506f9c6cfdd55d P 4ec1f2d4bde54666996929c96fa9a590 [term 2 FOLLOWER]: Committing config change with OpId 2.863: config changed from index 861 to 863, VOTER c616208720db4b14a9219cae8f2b7d49 (127.27.196.67) evicted. New config: { opid_index: 863 OBSOLETE_local: false peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } } }
I20240502 21:28:29.178905  2045 catalog_manager.cc:5484] T d4392d14cf254b72bf506f9c6cfdd55d P 4e8c16d1a1484a41a54b5ddc198becd3 reported cstate change: config changed from index 861 to 863, VOTER c616208720db4b14a9219cae8f2b7d49 (127.27.196.67) evicted. New cstate: current_term: 2 leader_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" committed_config { opid_index: 863 OBSOLETE_local: false peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } } }
I20240502 21:28:29.185027  2447 tablet_service.cc:1514] Processing DeleteTablet for tablet d4392d14cf254b72bf506f9c6cfdd55d with delete_type TABLET_DATA_TOMBSTONED (TS c616208720db4b14a9219cae8f2b7d49 not found in new config with opid_index 863) from {username='slave'} at 127.0.0.1:40204
I20240502 21:28:29.187741  8632 tablet_replica.cc:330] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49: stopping tablet replica
I20240502 21:28:29.188095  8632 raft_consensus.cc:2229] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:29.188378  8632 raft_consensus.cc:2258] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:29.190084  8632 ts_tablet_manager.cc:1875] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:29.196308  8632 ts_tablet_manager.cc:1888] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.862
I20240502 21:28:29.197054  8632 log.cc:1196] T d4392d14cf254b72bf506f9c6cfdd55d P c616208720db4b14a9219cae8f2b7d49: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/wals/d4392d14cf254b72bf506f9c6cfdd55d
I20240502 21:28:29.198150  2037 catalog_manager.cc:4830] TS c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687): tablet d4392d14cf254b72bf506f9c6cfdd55d (table rebalance_test_table_2 [id=dbf5b79f029b456d964098be120a38fd]) successfully deleted
I20240502 21:28:29.304556  3798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:29.323328  2447 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:29.323971  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:29.324611  7684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:29.332509  3665 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:29.354393  2174 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:29.361189  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:29.397027  2308 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:29.532362  2193 consensus_queue.cc:237] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 924, Committed index: 924, Last appended: 2.924, Last appended by leader: 923, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 925 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } }
I20240502 21:28:29.535231  7704 raft_consensus.cc:1261] T 3a6da907ad0447b3ad2f66f50cd7289d P 4ec1f2d4bde54666996929c96fa9a590 [term 2 FOLLOWER]: Refusing update from remote peer d961985de7874e1990a19bb9b2da14c2: Log matching property violated. Preceding OpId in replica: term: 2 index: 924. Preceding OpId from leader: term: 2 index: 925. (index mismatch)
I20240502 21:28:29.536068  8552 consensus_queue.cc:1035] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 925, Last known committed idx: 924, Time since last communication: 0.000s
I20240502 21:28:29.540012  8685 raft_consensus.cc:2932] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2 [term 2 LEADER]: Committing config change with OpId 2.925: config changed from index 923 to 925, VOTER 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66) evicted. New config: { opid_index: 925 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } } }
I20240502 21:28:29.540866  7704 raft_consensus.cc:2932] T 3a6da907ad0447b3ad2f66f50cd7289d P 4ec1f2d4bde54666996929c96fa9a590 [term 2 FOLLOWER]: Committing config change with OpId 2.925: config changed from index 923 to 925, VOTER 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66) evicted. New config: { opid_index: 925 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } } }
I20240502 21:28:29.545300  2045 catalog_manager.cc:5484] T 3a6da907ad0447b3ad2f66f50cd7289d P d961985de7874e1990a19bb9b2da14c2 reported cstate change: config changed from index 923 to 925, VOTER 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66) evicted. New cstate: current_term: 2 leader_uuid: "d961985de7874e1990a19bb9b2da14c2" committed_config { opid_index: 925 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } } }
I20240502 21:28:29.550663  2308 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a6da907ad0447b3ad2f66f50cd7289d with delete_type TABLET_DATA_TOMBSTONED (TS 68f426685b4c4c90bda61ef52177cc22 not found in new config with opid_index 925) from {username='slave'} at 127.0.0.1:35492
I20240502 21:28:29.552050  8687 tablet_replica.cc:330] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22: stopping tablet replica
I20240502 21:28:29.552441  8687 raft_consensus.cc:2229] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:29.552688  8687 raft_consensus.cc:2258] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:29.554970  8687 ts_tablet_manager.cc:1875] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:29.563973  8687 ts_tablet_manager.cc:1888] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.924
I20240502 21:28:29.564182  8687 log.cc:1196] T 3a6da907ad0447b3ad2f66f50cd7289d P 68f426685b4c4c90bda61ef52177cc22: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/wals/3a6da907ad0447b3ad2f66f50cd7289d
I20240502 21:28:29.565887  2036 catalog_manager.cc:4830] TS 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253): tablet 3a6da907ad0447b3ad2f66f50cd7289d (table rebalance_test_table_3 [id=72576758fa73429dba7089fdaa3f9ea3]) successfully deleted
I20240502 21:28:29.696655  3798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:29.714566  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:29.715401  3665 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:29.719556  2308 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:29.721721  7684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:29.751812  2447 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:29.764562  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:29.802417  2174 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:29.903978  3814 consensus_queue.cc:237] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1102, Committed index: 1102, Last appended: 2.1102, Last appended by leader: 1101, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1103 OBSOLETE_local: false peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } }
I20240502 21:28:29.906008  7704 raft_consensus.cc:1261] T 068c9a3114f648d1a183c9b90f4d365e P 4ec1f2d4bde54666996929c96fa9a590 [term 2 FOLLOWER]: Refusing update from remote peer 1193f1d0d2c843139d98458cf4035eb8: Log matching property violated. Preceding OpId in replica: term: 2 index: 1102. Preceding OpId from leader: term: 2 index: 1103. (index mismatch)
I20240502 21:28:29.906704  8557 consensus_queue.cc:1035] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1103, Last known committed idx: 1102, Time since last communication: 0.000s
I20240502 21:28:29.910153  8747 raft_consensus.cc:2932] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8 [term 2 LEADER]: Committing config change with OpId 2.1103: config changed from index 1101 to 1103, VOTER d961985de7874e1990a19bb9b2da14c2 (127.27.196.65) evicted. New config: { opid_index: 1103 OBSOLETE_local: false peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } } }
I20240502 21:28:29.910878  7704 raft_consensus.cc:2932] T 068c9a3114f648d1a183c9b90f4d365e P 4ec1f2d4bde54666996929c96fa9a590 [term 2 FOLLOWER]: Committing config change with OpId 2.1103: config changed from index 1101 to 1103, VOTER d961985de7874e1990a19bb9b2da14c2 (127.27.196.65) evicted. New config: { opid_index: 1103 OBSOLETE_local: false peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } } }
I20240502 21:28:29.916438  2046 catalog_manager.cc:5484] T 068c9a3114f648d1a183c9b90f4d365e P 1193f1d0d2c843139d98458cf4035eb8 reported cstate change: config changed from index 1101 to 1103, VOTER d961985de7874e1990a19bb9b2da14c2 (127.27.196.65) evicted. New cstate: current_term: 2 leader_uuid: "1193f1d0d2c843139d98458cf4035eb8" committed_config { opid_index: 1103 OBSOLETE_local: false peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } } }
I20240502 21:28:29.921960  2174 tablet_service.cc:1514] Processing DeleteTablet for tablet 068c9a3114f648d1a183c9b90f4d365e with delete_type TABLET_DATA_TOMBSTONED (TS d961985de7874e1990a19bb9b2da14c2 not found in new config with opid_index 1103) from {username='slave'} at 127.0.0.1:41124
I20240502 21:28:29.922794  8749 tablet_replica.cc:330] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2: stopping tablet replica
I20240502 21:28:29.923204  8749 raft_consensus.cc:2229] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:29.923566  8749 raft_consensus.cc:2258] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:29.927417  8749 ts_tablet_manager.cc:1875] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:29.940039  8749 ts_tablet_manager.cc:1888] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1102
I20240502 21:28:29.940265  8749 log.cc:1196] T 068c9a3114f648d1a183c9b90f4d365e P d961985de7874e1990a19bb9b2da14c2: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/wals/068c9a3114f648d1a183c9b90f4d365e
I20240502 21:28:29.941659  2038 catalog_manager.cc:4830] TS d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025): tablet 068c9a3114f648d1a183c9b90f4d365e (table rebalance_test_table_1 [id=2c42f3c4552d47169195e69073614d5b]) successfully deleted
I20240502 21:28:30.089532  3798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:30.090423  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:30.103406  3665 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:30.103565  2308 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:30.123878  2447 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:30.138094  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:30.151659  7684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:30.163596  2174 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:30.369803  3798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:30.391759  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:30.399211  7684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:30.419454  3665 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:30.432667  2308 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:30.493604  2447 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:30.495918  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:30.502391  2174 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:31.078763  3530 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:31.080489  3798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:31.095031  3665 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:31.112145  2447 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:28:31.113173  7684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:31.146219  3944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:31.149376  2308 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:31.180627  2174 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 b87cabdf7aa145babe0be5701e5c3815 | 127.27.196.126:43629 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                Value                                                                                 |         Tags          |         Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 allow_unsafe_replication_factor          | true                                                                                                                                                                 | runtime,unsafe        | all 1 server(s) checked
 ipki_ca_key_size                         | 768                                                                                                                                                                  | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                  | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                 | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                    | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | false                                                                                                                                                                | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                 | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                 | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                   | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                  | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |        Value        |         Master
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.27.196.84:40533 | all 1 server(s) checked
 time_source         | builtin             | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 1193f1d0d2c843139d98458cf4035eb8 | 127.27.196.70:46179 | HEALTHY | <none>   |       1        |       0
 13451be0f7134a2c89c0c9330177213e | 127.27.196.68:44575 | HEALTHY | <none>   |       2        |       0
 4e8c16d1a1484a41a54b5ddc198becd3 | 127.27.196.69:37783 | HEALTHY | <none>   |       3        |       0
 4ec1f2d4bde54666996929c96fa9a590 | 127.27.196.72:44723 | HEALTHY | <none>   |       0        |       0
 68f426685b4c4c90bda61ef52177cc22 | 127.27.196.66:36253 | HEALTHY | <none>   |       4        |       0
 c616208720db4b14a9219cae8f2b7d49 | 127.27.196.67:46687 | HEALTHY | <none>   |       2        |       0
 d306b26127e04e8c85a98edf1b41978a | 127.27.196.71:39583 | HEALTHY | <none>   |       0        |       0
 d961985de7874e1990a19bb9b2da14c2 | 127.27.196.65:37025 | HEALTHY | <none>   |       4        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       8

Unusual flags for Tablet Server:
                   Flag                   |                                                                              Value                                                                               |         Tags          |      Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                              | experimental          | all 8 server(s) checked
 local_ip_for_outbound_sockets            | 127.27.196.65                                                                                                                                                    | experimental          | 127.27.196.65:37025
 local_ip_for_outbound_sockets            | 127.27.196.66                                                                                                                                                    | experimental          | 127.27.196.66:36253
 local_ip_for_outbound_sockets            | 127.27.196.67                                                                                                                                                    | experimental          | 127.27.196.67:46687
 local_ip_for_outbound_sockets            | 127.27.196.68                                                                                                                                                    | experimental          | 127.27.196.68:44575
 local_ip_for_outbound_sockets            | 127.27.196.69                                                                                                                                                    | experimental          | 127.27.196.69:37783
 local_ip_for_outbound_sockets            | 127.27.196.70                                                                                                                                                    | experimental          | 127.27.196.70:46179
 local_ip_for_outbound_sockets            | 127.27.196.71                                                                                                                                                    | experimental          | 127.27.196.71:39583
 local_ip_for_outbound_sockets            | 127.27.196.72                                                                                                                                                    | experimental          | 127.27.196.72:44723
 never_fsync                              | true                                                                                                                                                             | unsafe,advanced       | all 8 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                | unsafe,hidden         | all 8 server(s) checked
 raft_prepare_replacement_before_eviction | false                                                                                                                                                            | experimental,advanced | all 8 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                             | unsafe                | all 8 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                               | hidden                | all 8 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.27.196.65:37025
 server_dump_info_path                    | /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.27.196.66:36253
 server_dump_info_path                    | /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.27.196.67:46687
 server_dump_info_path                    | /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.27.196.68:44575
 server_dump_info_path                    | /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden                | 127.27.196.69:37783
 server_dump_info_path                    | /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden                | 127.27.196.70:46179
 server_dump_info_path                    | /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden                | 127.27.196.71:39583
 server_dump_info_path                    | /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden                | 127.27.196.72:44723

Flags of checked categories for Tablet Server:
        Flag         |        Value        |      Tablet Server
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.27.196.84:40533 | all 8 server(s) checked
 time_source         | builtin             | all 8 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 9 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 2  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_4 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 4
 First Quartile | 4
 Median         | 4
 Third Quartile | 4
 Maximum        | 4

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 8
 Tables         | 6
 Tablets        | 16
 Replicas       | 32

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240502 21:28:31.242870 28433 log_verifier.cc:126] Checking tablet 0264c436aafa422a9a54599e74e0221c
I20240502 21:28:31.586941 28433 log_verifier.cc:177] Verified matching terms for 927 ops in tablet 0264c436aafa422a9a54599e74e0221c
I20240502 21:28:31.588636 28433 log_verifier.cc:126] Checking tablet 0306f5bfea0a49eca06554a40ba50d34
I20240502 21:28:31.990437 28433 log_verifier.cc:177] Verified matching terms for 970 ops in tablet 0306f5bfea0a49eca06554a40ba50d34
I20240502 21:28:31.991851 28433 log_verifier.cc:126] Checking tablet 068c9a3114f648d1a183c9b90f4d365e
I20240502 21:28:32.364506 28433 log_verifier.cc:177] Verified matching terms for 1103 ops in tablet 068c9a3114f648d1a183c9b90f4d365e
I20240502 21:28:32.365900 28433 log_verifier.cc:126] Checking tablet 2779949906ff486da691638de7374050
I20240502 21:28:32.765761 28433 log_verifier.cc:177] Verified matching terms for 998 ops in tablet 2779949906ff486da691638de7374050
I20240502 21:28:32.766973 28433 log_verifier.cc:126] Checking tablet 3a6da907ad0447b3ad2f66f50cd7289d
I20240502 21:28:33.099203 28433 log_verifier.cc:177] Verified matching terms for 925 ops in tablet 3a6da907ad0447b3ad2f66f50cd7289d
I20240502 21:28:33.100840 28433 log_verifier.cc:126] Checking tablet 5ae512d4d5874a909b1f4186c4b9785a
I20240502 21:28:33.112962 28433 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 5ae512d4d5874a909b1f4186c4b9785a
I20240502 21:28:33.113076 28433 log_verifier.cc:126] Checking tablet 6a38d2b4246e497fa91d2a0a8b127244
I20240502 21:28:33.432181 28433 log_verifier.cc:177] Verified matching terms for 908 ops in tablet 6a38d2b4246e497fa91d2a0a8b127244
I20240502 21:28:33.433555 28433 log_verifier.cc:126] Checking tablet 6e1c5a5cc57243d7bf6e212e3a32dfe3
I20240502 21:28:33.800298 28433 log_verifier.cc:177] Verified matching terms for 1013 ops in tablet 6e1c5a5cc57243d7bf6e212e3a32dfe3
I20240502 21:28:33.801816 28433 log_verifier.cc:126] Checking tablet 99b8f0399d884e41b009da4c3f51fa93
I20240502 21:28:34.187445 28433 log_verifier.cc:177] Verified matching terms for 1132 ops in tablet 99b8f0399d884e41b009da4c3f51fa93
I20240502 21:28:34.188767 28433 log_verifier.cc:126] Checking tablet be2cbcd79c1e40f49701271bb247f4d0
I20240502 21:28:34.552136 28433 log_verifier.cc:177] Verified matching terms for 956 ops in tablet be2cbcd79c1e40f49701271bb247f4d0
I20240502 21:28:34.553342 28433 log_verifier.cc:126] Checking tablet d4392d14cf254b72bf506f9c6cfdd55d
I20240502 21:28:34.845721 28433 log_verifier.cc:177] Verified matching terms for 863 ops in tablet d4392d14cf254b72bf506f9c6cfdd55d
I20240502 21:28:34.846712 28433 log_verifier.cc:126] Checking tablet d64bd5e7c22749078d82ad8da12d75dd
I20240502 21:28:35.032038  2041 maintenance_manager.cc:419] P b87cabdf7aa145babe0be5701e5c3815: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033399
I20240502 21:28:35.069284  2040 maintenance_manager.cc:643] P b87cabdf7aa145babe0be5701e5c3815: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.036s	user 0.026s	sys 0.008s Metrics: {"bytes_written":12612,"cfile_init":1,"dirs.queue_time_us":452,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":1265,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":939,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":25,"thread_start_us":661,"threads_started":2,"wal-append.queue_time_us":460}
I20240502 21:28:35.070751  2041 maintenance_manager.cc:419] P b87cabdf7aa145babe0be5701e5c3815: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 3288 bytes on disk
I20240502 21:28:35.071753  2040 maintenance_manager.cc:643] P b87cabdf7aa145babe0be5701e5c3815: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240502 21:28:35.198755 28433 log_verifier.cc:177] Verified matching terms for 946 ops in tablet d64bd5e7c22749078d82ad8da12d75dd
I20240502 21:28:35.200525 28433 log_verifier.cc:126] Checking tablet d7db9b73c0594885b383b059520414e9
I20240502 21:28:35.532864 28433 log_verifier.cc:177] Verified matching terms for 1000 ops in tablet d7db9b73c0594885b383b059520414e9
I20240502 21:28:35.534265 28433 log_verifier.cc:126] Checking tablet dc17a741596a4cb59692b7f13cf254e7
I20240502 21:28:35.892560 28433 log_verifier.cc:177] Verified matching terms for 1093 ops in tablet dc17a741596a4cb59692b7f13cf254e7
I20240502 21:28:35.894413 28433 log_verifier.cc:126] Checking tablet e93810d704674fc4ad4d96501fb5ea42
I20240502 21:28:36.235399 28433 log_verifier.cc:177] Verified matching terms for 901 ops in tablet e93810d704674fc4ad4d96501fb5ea42
I20240502 21:28:36.236420 28433 log_verifier.cc:126] Checking tablet fe578b007ccd496ba2f70cda8e90c91f
I20240502 21:28:36.566825 28433 log_verifier.cc:177] Verified matching terms for 968 ops in tablet fe578b007ccd496ba2f70cda8e90c91f
I20240502 21:28:36.571628 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 2112
I20240502 21:28:36.601976  2234 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:37.005941 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 2112
W20240502 21:28:37.078964  7638 connection.cc:535] server connection from 127.27.196.65:47469 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:28:37.079043  2401 connection.cc:535] server connection from 127.27.196.65:56845 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:28:37.079371 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 2245
I20240502 21:28:37.087662  2374 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:37.471889 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 2245
I20240502 21:28:37.538138 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 2383
I20240502 21:28:37.545100  2507 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:37.881436 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 2383
I20240502 21:28:37.962178 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 3466
I20240502 21:28:37.969092  3593 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:28:38.061025  3619 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.67:46687: connect: Connection refused (error 111)
W20240502 21:28:38.063275  3619 consensus_peers.cc:485] T 99b8f0399d884e41b009da4c3f51fa93 P 4e8c16d1a1484a41a54b5ddc198becd3 -> Peer c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687): Couldn't send request to peer c616208720db4b14a9219cae8f2b7d49. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.67:46687: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:38.192626  8780 raft_consensus.cc:482] T be2cbcd79c1e40f49701271bb247f4d0 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68f426685b4c4c90bda61ef52177cc22)
I20240502 21:28:38.192934  8780 raft_consensus.cc:504] T be2cbcd79c1e40f49701271bb247f4d0 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "4e8c16d1a1484a41a54b5ddc198becd3" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 37783 } }
I20240502 21:28:38.193948  8780 leader_election.cc:288] T be2cbcd79c1e40f49701271bb247f4d0 P 4e8c16d1a1484a41a54b5ddc198becd3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253)
W20240502 21:28:38.196375  3618 leader_election.cc:334] T be2cbcd79c1e40f49701271bb247f4d0 P 4e8c16d1a1484a41a54b5ddc198becd3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253): Network error: Client connection negotiation failed: client connection to 127.27.196.66:36253: connect: Connection refused (error 111)
I20240502 21:28:38.196602  3618 leader_election.cc:302] T be2cbcd79c1e40f49701271bb247f4d0 P 4e8c16d1a1484a41a54b5ddc198becd3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 4e8c16d1a1484a41a54b5ddc198becd3; no voters: 68f426685b4c4c90bda61ef52177cc22
I20240502 21:28:38.197060  8780 raft_consensus.cc:2726] T be2cbcd79c1e40f49701271bb247f4d0 P 4e8c16d1a1484a41a54b5ddc198becd3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:28:38.259577  8897 raft_consensus.cc:482] T 6a38d2b4246e497fa91d2a0a8b127244 P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68f426685b4c4c90bda61ef52177cc22)
I20240502 21:28:38.259850  8897 raft_consensus.cc:504] T 6a38d2b4246e497fa91d2a0a8b127244 P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 383 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } }
I20240502 21:28:38.260731  8897 leader_election.cc:288] T 6a38d2b4246e497fa91d2a0a8b127244 P 1193f1d0d2c843139d98458cf4035eb8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253)
W20240502 21:28:38.262496  3751 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.66:36253: connect: Connection refused (error 111)
W20240502 21:28:38.264204  3751 leader_election.cc:334] T 6a38d2b4246e497fa91d2a0a8b127244 P 1193f1d0d2c843139d98458cf4035eb8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253): Network error: Client connection negotiation failed: client connection to 127.27.196.66:36253: connect: Connection refused (error 111)
I20240502 21:28:38.264441  3751 leader_election.cc:302] T 6a38d2b4246e497fa91d2a0a8b127244 P 1193f1d0d2c843139d98458cf4035eb8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 1193f1d0d2c843139d98458cf4035eb8; no voters: 68f426685b4c4c90bda61ef52177cc22
I20240502 21:28:38.264955  8897 raft_consensus.cc:2726] T 6a38d2b4246e497fa91d2a0a8b127244 P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:28:38.269578 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 3466
I20240502 21:28:38.293733  8897 raft_consensus.cc:482] T d7db9b73c0594885b383b059520414e9 P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68f426685b4c4c90bda61ef52177cc22)
I20240502 21:28:38.294082  8897 raft_consensus.cc:504] T d7db9b73c0594885b383b059520414e9 P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 485 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "1193f1d0d2c843139d98458cf4035eb8" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 46179 } }
I20240502 21:28:38.294895  8897 leader_election.cc:288] T d7db9b73c0594885b383b059520414e9 P 1193f1d0d2c843139d98458cf4035eb8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253)
W20240502 21:28:38.296974  3751 leader_election.cc:334] T d7db9b73c0594885b383b059520414e9 P 1193f1d0d2c843139d98458cf4035eb8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253): Network error: Client connection negotiation failed: client connection to 127.27.196.66:36253: connect: Connection refused (error 111)
I20240502 21:28:38.297183  3751 leader_election.cc:302] T d7db9b73c0594885b383b059520414e9 P 1193f1d0d2c843139d98458cf4035eb8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 1193f1d0d2c843139d98458cf4035eb8; no voters: 68f426685b4c4c90bda61ef52177cc22
I20240502 21:28:38.297549  8897 raft_consensus.cc:2726] T d7db9b73c0594885b383b059520414e9 P 1193f1d0d2c843139d98458cf4035eb8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:28:38.326335  8904 raft_consensus.cc:482] T dc17a741596a4cb59692b7f13cf254e7 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68f426685b4c4c90bda61ef52177cc22)
I20240502 21:28:38.326565  8904 raft_consensus.cc:504] T dc17a741596a4cb59692b7f13cf254e7 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "68f426685b4c4c90bda61ef52177cc22" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 36253 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
W20240502 21:28:38.327045  3896 connection.cc:535] server connection from 127.27.196.68:37007 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:28:38.327603 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 3603
I20240502 21:28:38.329494  8904 leader_election.cc:288] T dc17a741596a4cb59692b7f13cf254e7 P d306b26127e04e8c85a98edf1b41978a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253)
W20240502 21:28:38.329870  3895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.66:36253: connect: Connection refused (error 111)
W20240502 21:28:38.332540  3895 leader_election.cc:334] T dc17a741596a4cb59692b7f13cf254e7 P d306b26127e04e8c85a98edf1b41978a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68f426685b4c4c90bda61ef52177cc22 (127.27.196.66:36253): Network error: Client connection negotiation failed: client connection to 127.27.196.66:36253: connect: Connection refused (error 111)
I20240502 21:28:38.332938  3895 leader_election.cc:302] T dc17a741596a4cb59692b7f13cf254e7 P d306b26127e04e8c85a98edf1b41978a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: d306b26127e04e8c85a98edf1b41978a; no voters: 68f426685b4c4c90bda61ef52177cc22
I20240502 21:28:38.333490  8904 raft_consensus.cc:2726] T dc17a741596a4cb59692b7f13cf254e7 P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:28:38.336555  3726 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:38.507746  8909 raft_consensus.cc:482] T 3a6da907ad0447b3ad2f66f50cd7289d P 4ec1f2d4bde54666996929c96fa9a590 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d961985de7874e1990a19bb9b2da14c2)
I20240502 21:28:38.508004  8909 raft_consensus.cc:504] T 3a6da907ad0447b3ad2f66f50cd7289d P 4ec1f2d4bde54666996929c96fa9a590 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 925 OBSOLETE_local: false peers { permanent_uuid: "d961985de7874e1990a19bb9b2da14c2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 37025 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } }
I20240502 21:28:38.515789  8909 leader_election.cc:288] T 3a6da907ad0447b3ad2f66f50cd7289d P 4ec1f2d4bde54666996929c96fa9a590 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025)
W20240502 21:28:38.516211  7637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:37025: connect: Connection refused (error 111)
W20240502 21:28:38.520504  7637 leader_election.cc:334] T 3a6da907ad0447b3ad2f66f50cd7289d P 4ec1f2d4bde54666996929c96fa9a590 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d961985de7874e1990a19bb9b2da14c2 (127.27.196.65:37025): Network error: Client connection negotiation failed: client connection to 127.27.196.65:37025: connect: Connection refused (error 111)
I20240502 21:28:38.520851  7637 leader_election.cc:302] T 3a6da907ad0447b3ad2f66f50cd7289d P 4ec1f2d4bde54666996929c96fa9a590 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 4ec1f2d4bde54666996929c96fa9a590; no voters: d961985de7874e1990a19bb9b2da14c2
I20240502 21:28:38.521576  8909 raft_consensus.cc:2726] T 3a6da907ad0447b3ad2f66f50cd7289d P 4ec1f2d4bde54666996929c96fa9a590 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 21:28:38.674451 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 3603
W20240502 21:28:38.733876  3754 connection.cc:535] server connection from 127.27.196.69:47583 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:28:38.734185 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 3736
I20240502 21:28:38.741351  3861 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:38.934303  8909 raft_consensus.cc:482] T 0306f5bfea0a49eca06554a40ba50d34 P 4ec1f2d4bde54666996929c96fa9a590 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c616208720db4b14a9219cae8f2b7d49)
I20240502 21:28:38.934665  8909 raft_consensus.cc:504] T 0306f5bfea0a49eca06554a40ba50d34 P 4ec1f2d4bde54666996929c96fa9a590 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 970 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "4ec1f2d4bde54666996929c96fa9a590" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 44723 } }
I20240502 21:28:38.935643  8909 leader_election.cc:288] T 0306f5bfea0a49eca06554a40ba50d34 P 4ec1f2d4bde54666996929c96fa9a590 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687)
W20240502 21:28:38.937285  7636 leader_election.cc:334] T 0306f5bfea0a49eca06554a40ba50d34 P 4ec1f2d4bde54666996929c96fa9a590 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687): Network error: Client connection negotiation failed: client connection to 127.27.196.67:46687: connect: Connection refused (error 111)
I20240502 21:28:38.937438  7636 leader_election.cc:302] T 0306f5bfea0a49eca06554a40ba50d34 P 4ec1f2d4bde54666996929c96fa9a590 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 4ec1f2d4bde54666996929c96fa9a590; no voters: c616208720db4b14a9219cae8f2b7d49
I20240502 21:28:38.937755  8909 raft_consensus.cc:2726] T 0306f5bfea0a49eca06554a40ba50d34 P 4ec1f2d4bde54666996929c96fa9a590 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:28:39.010015  8915 raft_consensus.cc:482] T d64bd5e7c22749078d82ad8da12d75dd P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c616208720db4b14a9219cae8f2b7d49)
I20240502 21:28:39.010229  8915 raft_consensus.cc:504] T d64bd5e7c22749078d82ad8da12d75dd P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "c616208720db4b14a9219cae8f2b7d49" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 46687 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
I20240502 21:28:39.010907  8915 leader_election.cc:288] T d64bd5e7c22749078d82ad8da12d75dd P d306b26127e04e8c85a98edf1b41978a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687)
W20240502 21:28:39.013310  3896 leader_election.cc:334] T d64bd5e7c22749078d82ad8da12d75dd P d306b26127e04e8c85a98edf1b41978a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c616208720db4b14a9219cae8f2b7d49 (127.27.196.67:46687): Network error: Client connection negotiation failed: client connection to 127.27.196.67:46687: connect: Connection refused (error 111)
I20240502 21:28:39.013447  3896 leader_election.cc:302] T d64bd5e7c22749078d82ad8da12d75dd P d306b26127e04e8c85a98edf1b41978a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: d306b26127e04e8c85a98edf1b41978a; no voters: c616208720db4b14a9219cae8f2b7d49
I20240502 21:28:39.013733  8915 raft_consensus.cc:2726] T d64bd5e7c22749078d82ad8da12d75dd P d306b26127e04e8c85a98edf1b41978a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:28:39.059695 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 3736
W20240502 21:28:39.124366  7637 connection.cc:535] server connection from 127.27.196.70:60179 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:28:39.124456  2037 connection.cc:535] server connection from 127.27.196.70:60891 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:28:39.124799 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 3871
I20240502 21:28:39.133844  4006 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:39.432436  8915 raft_consensus.cc:482] T 5ae512d4d5874a909b1f4186c4b9785a P d306b26127e04e8c85a98edf1b41978a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 13451be0f7134a2c89c0c9330177213e)
I20240502 21:28:39.432713  8915 raft_consensus.cc:504] T 5ae512d4d5874a909b1f4186c4b9785a P d306b26127e04e8c85a98edf1b41978a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13451be0f7134a2c89c0c9330177213e" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 44575 } } peers { permanent_uuid: "d306b26127e04e8c85a98edf1b41978a" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 39583 } }
I20240502 21:28:39.433122 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 3871
I20240502 21:28:39.495587 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 7618
I20240502 21:28:39.504292  7746 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:39.741065 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 7618
I20240502 21:28:39.785265 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 2021
I20240502 21:28:39.786623  2083 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:40.096364 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 2021
2024-05-02T21:28:40Z chronyd exiting
[       OK ] RebalanceParamTest.Rebalance/2 (126041 ms)
[----------] 1 test from RebalanceParamTest (126041 ms total)

[----------] 1 test from ConcurrentRebalancersTest
[ RUN      ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0
I20240502 21:28:40.250092 28433 test_util.cc:268] Using random seed: 1231700676
I20240502 21:28:40.252221 28433 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:28:40.252308 28433 ts_itest-base.cc:113] --------------
I20240502 21:28:40.252342 28433 ts_itest-base.cc:114] 8 tablet servers
I20240502 21:28:40.252427 28433 ts_itest-base.cc:115] 3 replicas per TS
I20240502 21:28:40.252483 28433 ts_itest-base.cc:116] --------------
2024-05-02T21:28:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:28:40Z Disabled control of system clock
I20240502 21:28:40.334833 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.196.126:34625
--webserver_interface=127.27.196.126
--webserver_port=0
--builtin_ntp_servers=127.27.196.84:38081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.196.126:34625
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240502 21:28:40.803382  8930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:40.803805  8930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:40.803951  8930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:40.812595  8930 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:28:40.812780  8930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:28:40.812880  8930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:40.812970  8930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:28:40.813051  8930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:28:40.823640  8930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:38081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.27.196.126:34625
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.196.126:34625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.27.196.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:40.825304  8930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:40.827577  8930 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:40.842952  8935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:40.843541  8936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:40.844285  8938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:40.845063  8930 server_base.cc:1034] running on GCE node
I20240502 21:28:40.846132  8930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:40.848026  8930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:40.849313  8930 hybrid_clock.cc:648] HybridClock initialized: now 1714685320849264 us; error 56 us; skew 500 ppm
I20240502 21:28:40.850126  8930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:40.854233  8930 webserver.cc:460] Webserver started at http://127.27.196.126:44661/ using document root <none> and password file <none>
I20240502 21:28:40.855360  8930 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:40.855530  8930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:40.855872  8930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:40.859200  8930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "7eba6d8fd1d94cf8b98795fe2505a077"
format_stamp: "Formatted at 2024-05-02 21:28:40 on dist-test-slave-3bs1"
I20240502 21:28:40.860129  8930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "7eba6d8fd1d94cf8b98795fe2505a077"
format_stamp: "Formatted at 2024-05-02 21:28:40 on dist-test-slave-3bs1"
I20240502 21:28:40.867022  8930 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.001s
I20240502 21:28:40.872781  8944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:28:40.875013  8930 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.004s	sys 0.001s
I20240502 21:28:40.875324  8930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7eba6d8fd1d94cf8b98795fe2505a077"
format_stamp: "Formatted at 2024-05-02 21:28:40 on dist-test-slave-3bs1"
I20240502 21:28:40.875598  8930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:40.894449  8930 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:28:40.895474  8930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:40.895792  8930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:40.918694  8930 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.126:34625
I20240502 21:28:40.918807  8998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.126:34625 every 8 connection(s)
I20240502 21:28:40.920468  8930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 21:28:40.922680 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 8930
I20240502 21:28:40.922921 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:28:40.925477  8999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:40.940759  8999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eba6d8fd1d94cf8b98795fe2505a077: Bootstrap starting.
I20240502 21:28:40.944329  8999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eba6d8fd1d94cf8b98795fe2505a077: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:40.945502  8999 log.cc:824] T 00000000000000000000000000000000 P 7eba6d8fd1d94cf8b98795fe2505a077: Log is configured to *not* fsync() on all Append() calls
I20240502 21:28:40.948812  8999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eba6d8fd1d94cf8b98795fe2505a077: No bootstrap required, opened a new log
I20240502 21:28:40.955946  8999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eba6d8fd1d94cf8b98795fe2505a077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eba6d8fd1d94cf8b98795fe2505a077" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 34625 } }
I20240502 21:28:40.956384  8999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7eba6d8fd1d94cf8b98795fe2505a077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:40.956504  8999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eba6d8fd1d94cf8b98795fe2505a077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eba6d8fd1d94cf8b98795fe2505a077, State: Initialized, Role: FOLLOWER
I20240502 21:28:40.957191  8999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eba6d8fd1d94cf8b98795fe2505a077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eba6d8fd1d94cf8b98795fe2505a077" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 34625 } }
I20240502 21:28:40.957433  8999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7eba6d8fd1d94cf8b98795fe2505a077 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:28:40.957583  8999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eba6d8fd1d94cf8b98795fe2505a077 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:28:40.957757  8999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eba6d8fd1d94cf8b98795fe2505a077 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:40.960075  8999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eba6d8fd1d94cf8b98795fe2505a077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eba6d8fd1d94cf8b98795fe2505a077" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 34625 } }
I20240502 21:28:40.960641  8999 leader_election.cc:302] T 00000000000000000000000000000000 P 7eba6d8fd1d94cf8b98795fe2505a077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7eba6d8fd1d94cf8b98795fe2505a077; no voters: 
I20240502 21:28:40.961300  8999 leader_election.cc:288] T 00000000000000000000000000000000 P 7eba6d8fd1d94cf8b98795fe2505a077 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 21:28:40.961463  9004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7eba6d8fd1d94cf8b98795fe2505a077 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:40.962141  9004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7eba6d8fd1d94cf8b98795fe2505a077 [term 1 LEADER]: Becoming Leader. State: Replica: 7eba6d8fd1d94cf8b98795fe2505a077, State: Running, Role: LEADER
I20240502 21:28:40.962777  9004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eba6d8fd1d94cf8b98795fe2505a077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eba6d8fd1d94cf8b98795fe2505a077" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 34625 } }
I20240502 21:28:40.963518  8999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eba6d8fd1d94cf8b98795fe2505a077 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:28:40.965760  9005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eba6d8fd1d94cf8b98795fe2505a077 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7eba6d8fd1d94cf8b98795fe2505a077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eba6d8fd1d94cf8b98795fe2505a077" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 34625 } } }
I20240502 21:28:40.966089  9005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eba6d8fd1d94cf8b98795fe2505a077 [sys.catalog]: This master's current role is: LEADER
I20240502 21:28:40.967940  9010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:28:40.967159  9006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eba6d8fd1d94cf8b98795fe2505a077 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eba6d8fd1d94cf8b98795fe2505a077. Latest consensus state: current_term: 1 leader_uuid: "7eba6d8fd1d94cf8b98795fe2505a077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eba6d8fd1d94cf8b98795fe2505a077" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 34625 } } }
I20240502 21:28:40.968314  9006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eba6d8fd1d94cf8b98795fe2505a077 [sys.catalog]: This master's current role is: LEADER
I20240502 21:28:40.974623  9010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:28:40.995453  9010 catalog_manager.cc:1347] Generated new cluster ID: 19e1f09937504bfab100352f25175895
I20240502 21:28:40.995613  9010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:28:41.010257  9010 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:28:41.011842  9010 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:28:41.027801  9010 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7eba6d8fd1d94cf8b98795fe2505a077: Generated new TSK 0
I20240502 21:28:41.028785  9010 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:28:41.043781 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.65:0
--local_ip_for_outbound_sockets=127.27.196.65
--webserver_interface=127.27.196.65
--webserver_port=0
--tserver_master_addrs=127.27.196.126:34625
--builtin_ntp_servers=127.27.196.84:38081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 21:28:41.481117  9023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:41.481456  9023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:41.481590  9023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:41.489580  9023 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:28:41.489734  9023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:41.489861  9023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.65
I20240502 21:28:41.501523  9023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:38081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.27.196.65
--webserver_port=0
--tserver_master_addrs=127.27.196.126:34625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.65
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:41.503151  9023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:41.505044  9023 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:41.519182  9029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:41.519867  9031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:41.520923  9028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:41.521605  9023 server_base.cc:1034] running on GCE node
I20240502 21:28:41.522344  9023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:41.523604  9023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:41.524857  9023 hybrid_clock.cc:648] HybridClock initialized: now 1714685321524719 us; error 127 us; skew 500 ppm
I20240502 21:28:41.525264  9023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:41.528698  9023 webserver.cc:460] Webserver started at http://127.27.196.65:45021/ using document root <none> and password file <none>
I20240502 21:28:41.529570  9023 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:41.529826  9023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:41.530232  9023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:41.533250  9023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "5543ddab5b484642b139b7c848932a58"
format_stamp: "Formatted at 2024-05-02 21:28:41 on dist-test-slave-3bs1"
I20240502 21:28:41.534119  9023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "5543ddab5b484642b139b7c848932a58"
format_stamp: "Formatted at 2024-05-02 21:28:41 on dist-test-slave-3bs1"
I20240502 21:28:41.540755  9023 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.001s
I20240502 21:28:41.546491  9037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:28:41.548815  9023 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.002s	sys 0.001s
I20240502 21:28:41.549113  9023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "5543ddab5b484642b139b7c848932a58"
format_stamp: "Formatted at 2024-05-02 21:28:41 on dist-test-slave-3bs1"
I20240502 21:28:41.549374  9023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:41.570716  9023 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:28:41.571853  9023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:41.572175  9023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:41.573624  9023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:41.575687  9023 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:41.575796  9023 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:28:41.575901  9023 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:41.575991  9023 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:28:41.619475  9023 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.65:42177
I20240502 21:28:41.619526  9152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.65:42177 every 8 connection(s)
I20240502 21:28:41.621425  9023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:28:41.624833 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 9023
I20240502 21:28:41.625084 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:28:41.628717 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.66:0
--local_ip_for_outbound_sockets=127.27.196.66
--webserver_interface=127.27.196.66
--webserver_port=0
--tserver_master_addrs=127.27.196.126:34625
--builtin_ntp_servers=127.27.196.84:38081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:28:41.643334  9153 heartbeater.cc:347] Connected to a master server at 127.27.196.126:34625
I20240502 21:28:41.643777  9153 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:41.644742  9153 heartbeater.cc:511] Master 127.27.196.126:34625 requested a full tablet report, sending...
I20240502 21:28:41.647696  8963 ts_manager.cc:194] Registered new tserver with Master: 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:28:41.649456  8963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.65:40861
W20240502 21:28:42.070114  9157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:42.070556  9157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:42.070749  9157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:42.079800  9157 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:28:42.079988  9157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:42.080173  9157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.66
I20240502 21:28:42.090737  9157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:38081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.27.196.66
--webserver_port=0
--tserver_master_addrs=127.27.196.126:34625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.66
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:42.092612  9157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:42.094789  9157 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:42.111827  9163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:42.112553  9162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:42.113227  9165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:42.114111  9157 server_base.cc:1034] running on GCE node
I20240502 21:28:42.115247  9157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:42.116868  9157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:42.118170  9157 hybrid_clock.cc:648] HybridClock initialized: now 1714685322118083 us; error 76 us; skew 500 ppm
I20240502 21:28:42.118691  9157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:42.122061  9157 webserver.cc:460] Webserver started at http://127.27.196.66:38883/ using document root <none> and password file <none>
I20240502 21:28:42.123023  9157 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:42.123205  9157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:42.123590  9157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:42.126783  9157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "525a093ea5bf40dc9bbf9dc39895d670"
format_stamp: "Formatted at 2024-05-02 21:28:42 on dist-test-slave-3bs1"
I20240502 21:28:42.127900  9157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "525a093ea5bf40dc9bbf9dc39895d670"
format_stamp: "Formatted at 2024-05-02 21:28:42 on dist-test-slave-3bs1"
I20240502 21:28:42.135648  9157 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.006s	sys 0.001s
I20240502 21:28:42.141852  9171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:28:42.144052  9157 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.002s	sys 0.000s
I20240502 21:28:42.144321  9157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "525a093ea5bf40dc9bbf9dc39895d670"
format_stamp: "Formatted at 2024-05-02 21:28:42 on dist-test-slave-3bs1"
I20240502 21:28:42.144606  9157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:42.167995  9157 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:28:42.168980  9157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:42.169287  9157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:42.170630  9157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:42.172636  9157 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:42.172767  9157 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:28:42.172905  9157 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:42.172986  9157 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:28:42.216766  9157 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.66:39071
I20240502 21:28:42.216807  9288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.66:39071 every 8 connection(s)
I20240502 21:28:42.218850  9157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:28:42.219612 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 9157
I20240502 21:28:42.219846 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:28:42.224709 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.67:0
--local_ip_for_outbound_sockets=127.27.196.67
--webserver_interface=127.27.196.67
--webserver_port=0
--tserver_master_addrs=127.27.196.126:34625
--builtin_ntp_servers=127.27.196.84:38081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:28:42.240931  9289 heartbeater.cc:347] Connected to a master server at 127.27.196.126:34625
I20240502 21:28:42.241536  9289 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:42.243075  9289 heartbeater.cc:511] Master 127.27.196.126:34625 requested a full tablet report, sending...
I20240502 21:28:42.245954  8963 ts_manager.cc:194] Registered new tserver with Master: 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:28:42.247103  8963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.66:55281
I20240502 21:28:42.653174  9153 heartbeater.cc:503] Master 127.27.196.126:34625 was elected leader, sending a full tablet report...
W20240502 21:28:42.680532  9293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:42.680899  9293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:42.681015  9293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:42.688999  9293 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:28:42.689153  9293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:42.689282  9293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.67
I20240502 21:28:42.699003  9293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:38081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.27.196.67
--webserver_port=0
--tserver_master_addrs=127.27.196.126:34625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.67
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:42.701205  9293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:42.703752  9293 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:42.721194  9298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:42.721968  9299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:42.722420  9301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:42.723224  9293 server_base.cc:1034] running on GCE node
I20240502 21:28:42.724148  9293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:42.725297  9293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:42.726574  9293 hybrid_clock.cc:648] HybridClock initialized: now 1714685322726495 us; error 48 us; skew 500 ppm
I20240502 21:28:42.726955  9293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:42.729676  9293 webserver.cc:460] Webserver started at http://127.27.196.67:46445/ using document root <none> and password file <none>
I20240502 21:28:42.730523  9293 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:42.730660  9293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:42.731014  9293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:42.734017  9293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
format_stamp: "Formatted at 2024-05-02 21:28:42 on dist-test-slave-3bs1"
I20240502 21:28:42.734943  9293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
format_stamp: "Formatted at 2024-05-02 21:28:42 on dist-test-slave-3bs1"
I20240502 21:28:42.741948  9293 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.004s	sys 0.000s
I20240502 21:28:42.746681  9309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:28:42.748272  9293 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 21:28:42.748478  9293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
format_stamp: "Formatted at 2024-05-02 21:28:42 on dist-test-slave-3bs1"
I20240502 21:28:42.748723  9293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:42.762760  9293 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:28:42.763873  9293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:42.764182  9293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:42.765326  9293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:42.767024  9293 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:42.767150  9293 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:28:42.767277  9293 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:42.767357  9293 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:28:42.807021  9293 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.67:43203
I20240502 21:28:42.807090  9424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.67:43203 every 8 connection(s)
I20240502 21:28:42.808863  9293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:28:42.810074 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 9293
I20240502 21:28:42.810323 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:28:42.814810 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.68:0
--local_ip_for_outbound_sockets=127.27.196.68
--webserver_interface=127.27.196.68
--webserver_port=0
--tserver_master_addrs=127.27.196.126:34625
--builtin_ntp_servers=127.27.196.84:38081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:28:42.825615  9425 heartbeater.cc:347] Connected to a master server at 127.27.196.126:34625
I20240502 21:28:42.826015  9425 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:42.826879  9425 heartbeater.cc:511] Master 127.27.196.126:34625 requested a full tablet report, sending...
I20240502 21:28:42.828662  8963 ts_manager.cc:194] Registered new tserver with Master: f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:28:42.829946  8963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.67:37185
W20240502 21:28:43.244047  9429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:43.244421  9429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:43.244556  9429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 21:28:43.252089  9289 heartbeater.cc:503] Master 127.27.196.126:34625 was elected leader, sending a full tablet report...
W20240502 21:28:43.254900  9429 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:28:43.255156  9429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:43.255360  9429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.68
I20240502 21:28:43.268124  9429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:38081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.27.196.68
--webserver_port=0
--tserver_master_addrs=127.27.196.126:34625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.68
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:43.269805  9429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:43.271948  9429 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:43.287082  9434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:43.287739  9435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:43.288396  9437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:43.289294  9429 server_base.cc:1034] running on GCE node
I20240502 21:28:43.290138  9429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:43.291357  9429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:43.292639  9429 hybrid_clock.cc:648] HybridClock initialized: now 1714685323292599 us; error 37 us; skew 500 ppm
I20240502 21:28:43.293018  9429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:43.296123  9429 webserver.cc:460] Webserver started at http://127.27.196.68:42307/ using document root <none> and password file <none>
I20240502 21:28:43.296999  9429 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:43.297120  9429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:43.297438  9429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:43.300472  9429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "19f162e164854f2da623a4bb8f9e6bf6"
format_stamp: "Formatted at 2024-05-02 21:28:43 on dist-test-slave-3bs1"
I20240502 21:28:43.301366  9429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "19f162e164854f2da623a4bb8f9e6bf6"
format_stamp: "Formatted at 2024-05-02 21:28:43 on dist-test-slave-3bs1"
I20240502 21:28:43.307992  9429 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.004s	sys 0.004s
I20240502 21:28:43.312995  9443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:28:43.315810  9429 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.000s
I20240502 21:28:43.316126  9429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "19f162e164854f2da623a4bb8f9e6bf6"
format_stamp: "Formatted at 2024-05-02 21:28:43 on dist-test-slave-3bs1"
I20240502 21:28:43.316439  9429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:43.346813  9429 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:28:43.347925  9429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:43.348240  9429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:43.349459  9429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:43.351572  9429 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:43.351673  9429 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:28:43.351804  9429 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:43.351897  9429 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:28:43.393172  9429 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.68:40931
I20240502 21:28:43.393224  9560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.68:40931 every 8 connection(s)
I20240502 21:28:43.394827  9429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:28:43.404731 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 9429
I20240502 21:28:43.405109 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 21:28:43.410059 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.69:0
--local_ip_for_outbound_sockets=127.27.196.69
--webserver_interface=127.27.196.69
--webserver_port=0
--tserver_master_addrs=127.27.196.126:34625
--builtin_ntp_servers=127.27.196.84:38081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:28:43.415234  9561 heartbeater.cc:347] Connected to a master server at 127.27.196.126:34625
I20240502 21:28:43.415649  9561 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:43.416566  9561 heartbeater.cc:511] Master 127.27.196.126:34625 requested a full tablet report, sending...
I20240502 21:28:43.418377  8963 ts_manager.cc:194] Registered new tserver with Master: 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
I20240502 21:28:43.419382  8963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.68:50603
I20240502 21:28:43.833138  9425 heartbeater.cc:503] Master 127.27.196.126:34625 was elected leader, sending a full tablet report...
W20240502 21:28:43.852690  9565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:43.853032  9565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:43.853145  9565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:43.862433  9565 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:28:43.862641  9565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:43.862881  9565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.69
I20240502 21:28:43.873844  9565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:38081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.27.196.69
--webserver_port=0
--tserver_master_addrs=127.27.196.126:34625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.69
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:43.876073  9565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:43.879004  9565 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:43.897341  9570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:43.897765  9573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:43.898474  9571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:43.899439  9565 server_base.cc:1034] running on GCE node
I20240502 21:28:43.900486  9565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:43.902053  9565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:43.903581  9565 hybrid_clock.cc:648] HybridClock initialized: now 1714685323903452 us; error 95 us; skew 500 ppm
I20240502 21:28:43.904124  9565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:43.907469  9565 webserver.cc:460] Webserver started at http://127.27.196.69:45843/ using document root <none> and password file <none>
I20240502 21:28:43.908382  9565 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:43.908502  9565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:43.908844  9565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:43.911942  9565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "d10e9e156ba74a97a221c85b68ab8bc4"
format_stamp: "Formatted at 2024-05-02 21:28:43 on dist-test-slave-3bs1"
I20240502 21:28:43.912846  9565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "d10e9e156ba74a97a221c85b68ab8bc4"
format_stamp: "Formatted at 2024-05-02 21:28:43 on dist-test-slave-3bs1"
I20240502 21:28:43.919978  9565 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.006s	sys 0.001s
I20240502 21:28:43.924654  9579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:28:43.926677  9565 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.001s
I20240502 21:28:43.926904  9565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "d10e9e156ba74a97a221c85b68ab8bc4"
format_stamp: "Formatted at 2024-05-02 21:28:43 on dist-test-slave-3bs1"
I20240502 21:28:43.927189  9565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:43.973346  9565 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:28:43.974545  9565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:43.974897  9565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:43.976929  9565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:43.979720  9565 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:43.979919  9565 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:28:43.980087  9565 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:43.980180  9565 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:28:44.025650  9565 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.69:42049
I20240502 21:28:44.025713  9694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.69:42049 every 8 connection(s)
I20240502 21:28:44.028052  9565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:28:44.033428 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 9565
I20240502 21:28:44.033645 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 21:28:44.037576 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.70:0
--local_ip_for_outbound_sockets=127.27.196.70
--webserver_interface=127.27.196.70
--webserver_port=0
--tserver_master_addrs=127.27.196.126:34625
--builtin_ntp_servers=127.27.196.84:38081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:28:44.046622  9695 heartbeater.cc:347] Connected to a master server at 127.27.196.126:34625
I20240502 21:28:44.047046  9695 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:44.048058  9695 heartbeater.cc:511] Master 127.27.196.126:34625 requested a full tablet report, sending...
I20240502 21:28:44.049810  8963 ts_manager.cc:194] Registered new tserver with Master: d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
I20240502 21:28:44.050832  8963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.69:48097
I20240502 21:28:44.422526  9561 heartbeater.cc:503] Master 127.27.196.126:34625 was elected leader, sending a full tablet report...
W20240502 21:28:44.486151  9699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:44.486611  9699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:44.486752  9699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:44.496162  9699 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:28:44.496347  9699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:44.496562  9699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.70
I20240502 21:28:44.506676  9699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:38081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.27.196.70
--webserver_port=0
--tserver_master_addrs=127.27.196.126:34625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.70
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:44.508666  9699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:44.510903  9699 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:44.528299  9704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:44.528909  9705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:44.529732  9707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:44.530618  9699 server_base.cc:1034] running on GCE node
I20240502 21:28:44.531471  9699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:44.532722  9699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:44.534001  9699 hybrid_clock.cc:648] HybridClock initialized: now 1714685324533885 us; error 79 us; skew 500 ppm
I20240502 21:28:44.534694  9699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:44.538784  9699 webserver.cc:460] Webserver started at http://127.27.196.70:43831/ using document root <none> and password file <none>
I20240502 21:28:44.540084  9699 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:44.540261  9699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:44.540750  9699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:44.544982  9699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "6ef0beaf2f6941afb6ea9f9f274517fe"
format_stamp: "Formatted at 2024-05-02 21:28:44 on dist-test-slave-3bs1"
I20240502 21:28:44.546039  9699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "6ef0beaf2f6941afb6ea9f9f274517fe"
format_stamp: "Formatted at 2024-05-02 21:28:44 on dist-test-slave-3bs1"
I20240502 21:28:44.553123  9699 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.004s	sys 0.004s
I20240502 21:28:44.557631  9713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:28:44.559656  9699 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.001s
I20240502 21:28:44.559906  9699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6ef0beaf2f6941afb6ea9f9f274517fe"
format_stamp: "Formatted at 2024-05-02 21:28:44 on dist-test-slave-3bs1"
I20240502 21:28:44.560161  9699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:44.586508  9699 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:28:44.587538  9699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:44.587815  9699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:44.589130  9699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:44.591048  9699 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:44.591177  9699 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.001s	sys 0.000s
I20240502 21:28:44.591274  9699 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:44.591357  9699 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:28:44.633245  9699 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.70:44211
I20240502 21:28:44.633350  9827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.70:44211 every 8 connection(s)
I20240502 21:28:44.635223  9699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 21:28:44.642542 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 9699
I20240502 21:28:44.642839 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 21:28:44.647768 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.71:0
--local_ip_for_outbound_sockets=127.27.196.71
--webserver_interface=127.27.196.71
--webserver_port=0
--tserver_master_addrs=127.27.196.126:34625
--builtin_ntp_servers=127.27.196.84:38081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:28:44.658047  9828 heartbeater.cc:347] Connected to a master server at 127.27.196.126:34625
I20240502 21:28:44.658466  9828 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:44.659502  9828 heartbeater.cc:511] Master 127.27.196.126:34625 requested a full tablet report, sending...
I20240502 21:28:44.662006  8963 ts_manager.cc:194] Registered new tserver with Master: 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211)
I20240502 21:28:44.663230  8963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.70:50573
I20240502 21:28:45.053943  9695 heartbeater.cc:503] Master 127.27.196.126:34625 was elected leader, sending a full tablet report...
W20240502 21:28:45.099009  9832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:45.099608  9832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:45.099818  9832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:45.108054  9832 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:28:45.108242  9832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:45.108445  9832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.71
I20240502 21:28:45.118220  9832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:38081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.27.196.71
--webserver_port=0
--tserver_master_addrs=127.27.196.126:34625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.71
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:45.119923  9832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:45.122118  9832 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:45.140401  9840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:45.140627  9839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:45.141453  9842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:45.142160  9832 server_base.cc:1034] running on GCE node
I20240502 21:28:45.143090  9832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:45.144724  9832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:45.146023  9832 hybrid_clock.cc:648] HybridClock initialized: now 1714685325145977 us; error 43 us; skew 500 ppm
I20240502 21:28:45.146476  9832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:45.149470  9832 webserver.cc:460] Webserver started at http://127.27.196.71:39053/ using document root <none> and password file <none>
I20240502 21:28:45.150846  9832 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:45.150974  9832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:45.151451  9832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:45.154915  9832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "4cbb5b06a6a44eb6b2145cd538b01af2"
format_stamp: "Formatted at 2024-05-02 21:28:45 on dist-test-slave-3bs1"
I20240502 21:28:45.156039  9832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "4cbb5b06a6a44eb6b2145cd538b01af2"
format_stamp: "Formatted at 2024-05-02 21:28:45 on dist-test-slave-3bs1"
I20240502 21:28:45.163354  9832 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.007s	sys 0.001s
I20240502 21:28:45.168882  9848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:28:45.171197  9832 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.005s	sys 0.000s
I20240502 21:28:45.171447  9832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "4cbb5b06a6a44eb6b2145cd538b01af2"
format_stamp: "Formatted at 2024-05-02 21:28:45 on dist-test-slave-3bs1"
I20240502 21:28:45.171695  9832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:45.189563  9832 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:28:45.190629  9832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:45.190932  9832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:45.192195  9832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:45.194124  9832 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:45.194228  9832 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:28:45.194339  9832 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:45.194424  9832 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:28:45.233547  9832 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.71:35857
I20240502 21:28:45.233603  9966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.71:35857 every 8 connection(s)
I20240502 21:28:45.235172  9832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 21:28:45.237322 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 9832
I20240502 21:28:45.237551 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 21:28:45.242568 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.72:0
--local_ip_for_outbound_sockets=127.27.196.72
--webserver_interface=127.27.196.72
--webserver_port=0
--tserver_master_addrs=127.27.196.126:34625
--builtin_ntp_servers=127.27.196.84:38081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:28:45.253314  9967 heartbeater.cc:347] Connected to a master server at 127.27.196.126:34625
I20240502 21:28:45.253996  9967 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:45.255548  9967 heartbeater.cc:511] Master 127.27.196.126:34625 requested a full tablet report, sending...
I20240502 21:28:45.258500  8963 ts_manager.cc:194] Registered new tserver with Master: 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857)
I20240502 21:28:45.259935  8963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.71:49217
W20240502 21:28:45.667404  9971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:45.667917  9971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:45.668118  9971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 21:28:45.668568  9828 heartbeater.cc:503] Master 127.27.196.126:34625 was elected leader, sending a full tablet report...
W20240502 21:28:45.678308  9971 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:28:45.678500  9971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:45.678664  9971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.72
I20240502 21:28:45.690328  9971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:38081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.27.196.72
--webserver_port=0
--tserver_master_addrs=127.27.196.126:34625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.72
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:45.692011  9971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:45.695191  9971 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:45.712271  9977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:45.712936  9978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:45.713713  9980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:45.714640  9971 server_base.cc:1034] running on GCE node
I20240502 21:28:45.715502  9971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:45.716784  9971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:45.718286  9971 hybrid_clock.cc:648] HybridClock initialized: now 1714685325718194 us; error 84 us; skew 500 ppm
I20240502 21:28:45.719040  9971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:45.722885  9971 webserver.cc:460] Webserver started at http://127.27.196.72:33717/ using document root <none> and password file <none>
I20240502 21:28:45.724342  9971 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:45.724504  9971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:45.724987  9971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:45.728610  9971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "e0de0545fa0547d3929111e990e6d1b2"
format_stamp: "Formatted at 2024-05-02 21:28:45 on dist-test-slave-3bs1"
I20240502 21:28:45.729578  9971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "e0de0545fa0547d3929111e990e6d1b2"
format_stamp: "Formatted at 2024-05-02 21:28:45 on dist-test-slave-3bs1"
I20240502 21:28:45.736929  9971 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.008s	sys 0.000s
I20240502 21:28:45.741832  9986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:28:45.743906  9971 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 21:28:45.744138  9971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "e0de0545fa0547d3929111e990e6d1b2"
format_stamp: "Formatted at 2024-05-02 21:28:45 on dist-test-slave-3bs1"
I20240502 21:28:45.744382  9971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:45.763796  9971 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:28:45.764850  9971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:45.765187  9971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:45.766633  9971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:45.768599  9971 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:45.768704  9971 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:28:45.768826  9971 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:45.768899  9971 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:28:45.810354  9971 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.72:40467
I20240502 21:28:45.810434 10103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.72:40467 every 8 connection(s)
I20240502 21:28:45.811884  9971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 21:28:45.815663 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 9971
I20240502 21:28:45.815910 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 21:28:45.828014 10104 heartbeater.cc:347] Connected to a master server at 127.27.196.126:34625
I20240502 21:28:45.828410 10104 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:45.829334 10104 heartbeater.cc:511] Master 127.27.196.126:34625 requested a full tablet report, sending...
I20240502 21:28:45.831300  8962 ts_manager.cc:194] Registered new tserver with Master: e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467)
I20240502 21:28:45.832672  8962 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.72:47129
I20240502 21:28:45.834547 28433 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 21:28:45.858012 28433 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 21:28:45.858207 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 9565
I20240502 21:28:45.869246  9690 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:45.961763 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 9565
I20240502 21:28:45.983983 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 9699
I20240502 21:28:45.994107  9823 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:46.085366 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 9699
I20240502 21:28:46.106388 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 9832
I20240502 21:28:46.117574  9962 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:46.209952 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 9832
I20240502 21:28:46.233697 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 9971
I20240502 21:28:46.245545 10099 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:46.335479 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 9971
I20240502 21:28:50.113912  8962 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54676:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 21:28:50.157316  9356 tablet_service.cc:1467] Processing CreateTablet for tablet 41ee481a3ece47a0905e93c66dbf949d (DEFAULT_TABLE table=rebalance_test_table_0 [id=445d2e6ec7154b80abec5fff346b8f5e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:50.159641  9356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ee481a3ece47a0905e93c66dbf949d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:50.157361  9357 tablet_service.cc:1467] Processing CreateTablet for tablet de3287296c4547688388388dd4c7e638 (DEFAULT_TABLE table=rebalance_test_table_0 [id=445d2e6ec7154b80abec5fff346b8f5e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:50.163759  9357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de3287296c4547688388388dd4c7e638. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:50.168067  9490 tablet_service.cc:1467] Processing CreateTablet for tablet 9240163cfac446d2a14dbd9fcbbf9ff7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=445d2e6ec7154b80abec5fff346b8f5e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:50.169471  9220 tablet_service.cc:1467] Processing CreateTablet for tablet 9240163cfac446d2a14dbd9fcbbf9ff7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=445d2e6ec7154b80abec5fff346b8f5e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:50.170074  9490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9240163cfac446d2a14dbd9fcbbf9ff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:50.168170  9491 tablet_service.cc:1467] Processing CreateTablet for tablet de3287296c4547688388388dd4c7e638 (DEFAULT_TABLE table=rebalance_test_table_0 [id=445d2e6ec7154b80abec5fff346b8f5e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:50.170903  9220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9240163cfac446d2a14dbd9fcbbf9ff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:50.172266  9221 tablet_service.cc:1467] Processing CreateTablet for tablet 41ee481a3ece47a0905e93c66dbf949d (DEFAULT_TABLE table=rebalance_test_table_0 [id=445d2e6ec7154b80abec5fff346b8f5e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:50.176470  9087 tablet_service.cc:1467] Processing CreateTablet for tablet de3287296c4547688388388dd4c7e638 (DEFAULT_TABLE table=rebalance_test_table_0 [id=445d2e6ec7154b80abec5fff346b8f5e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:50.178555  9087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de3287296c4547688388388dd4c7e638. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:50.178653  9221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ee481a3ece47a0905e93c66dbf949d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:50.179289  9491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de3287296c4547688388388dd4c7e638. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:50.187439  9086 tablet_service.cc:1467] Processing CreateTablet for tablet 41ee481a3ece47a0905e93c66dbf949d (DEFAULT_TABLE table=rebalance_test_table_0 [id=445d2e6ec7154b80abec5fff346b8f5e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:50.188170  9086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ee481a3ece47a0905e93c66dbf949d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:50.189865 10135 tablet_bootstrap.cc:492] T 41ee481a3ece47a0905e93c66dbf949d P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap starting.
I20240502 21:28:50.191545  9085 tablet_service.cc:1467] Processing CreateTablet for tablet 9240163cfac446d2a14dbd9fcbbf9ff7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=445d2e6ec7154b80abec5fff346b8f5e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:50.192250  9085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9240163cfac446d2a14dbd9fcbbf9ff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:50.194762 10135 tablet_bootstrap.cc:654] T 41ee481a3ece47a0905e93c66dbf949d P 525a093ea5bf40dc9bbf9dc39895d670: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:50.196440 10135 log.cc:824] T 41ee481a3ece47a0905e93c66dbf949d P 525a093ea5bf40dc9bbf9dc39895d670: Log is configured to *not* fsync() on all Append() calls
I20240502 21:28:50.201406 10135 tablet_bootstrap.cc:492] T 41ee481a3ece47a0905e93c66dbf949d P 525a093ea5bf40dc9bbf9dc39895d670: No bootstrap required, opened a new log
I20240502 21:28:50.201894 10135 ts_tablet_manager.cc:1367] T 41ee481a3ece47a0905e93c66dbf949d P 525a093ea5bf40dc9bbf9dc39895d670: Time spent bootstrapping tablet: real 0.012s	user 0.011s	sys 0.000s
I20240502 21:28:50.215853 10135 raft_consensus.cc:348] T 41ee481a3ece47a0905e93c66dbf949d P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:50.217370 10138 tablet_bootstrap.cc:492] T 41ee481a3ece47a0905e93c66dbf949d P 5543ddab5b484642b139b7c848932a58: Bootstrap starting.
I20240502 21:28:50.220002 10139 tablet_bootstrap.cc:492] T de3287296c4547688388388dd4c7e638 P 19f162e164854f2da623a4bb8f9e6bf6: Bootstrap starting.
I20240502 21:28:50.221417 10138 tablet_bootstrap.cc:654] T 41ee481a3ece47a0905e93c66dbf949d P 5543ddab5b484642b139b7c848932a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:50.222132 10134 tablet_bootstrap.cc:492] T de3287296c4547688388388dd4c7e638 P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap starting.
I20240502 21:28:50.222484 10135 raft_consensus.cc:374] T 41ee481a3ece47a0905e93c66dbf949d P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:50.222707 10135 raft_consensus.cc:729] T 41ee481a3ece47a0905e93c66dbf949d P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Initialized, Role: FOLLOWER
I20240502 21:28:50.225095 10135 consensus_queue.cc:260] T 41ee481a3ece47a0905e93c66dbf949d P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:50.226341 10138 log.cc:824] T 41ee481a3ece47a0905e93c66dbf949d P 5543ddab5b484642b139b7c848932a58: Log is configured to *not* fsync() on all Append() calls
I20240502 21:28:50.226495 10134 tablet_bootstrap.cc:654] T de3287296c4547688388388dd4c7e638 P f2fdd596a8b44bbc848fe40f9f07fbe0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:50.227078 10139 tablet_bootstrap.cc:654] T de3287296c4547688388388dd4c7e638 P 19f162e164854f2da623a4bb8f9e6bf6: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:50.227952 10134 log.cc:824] T de3287296c4547688388388dd4c7e638 P f2fdd596a8b44bbc848fe40f9f07fbe0: Log is configured to *not* fsync() on all Append() calls
I20240502 21:28:50.229601 10139 log.cc:824] T de3287296c4547688388388dd4c7e638 P 19f162e164854f2da623a4bb8f9e6bf6: Log is configured to *not* fsync() on all Append() calls
I20240502 21:28:50.230717 10135 ts_tablet_manager.cc:1398] T 41ee481a3ece47a0905e93c66dbf949d P 525a093ea5bf40dc9bbf9dc39895d670: Time spent starting tablet: real 0.029s	user 0.016s	sys 0.000s
I20240502 21:28:50.231359 10135 tablet_bootstrap.cc:492] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap starting.
I20240502 21:28:50.233481 10135 tablet_bootstrap.cc:654] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 525a093ea5bf40dc9bbf9dc39895d670: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:50.233577 10134 tablet_bootstrap.cc:492] T de3287296c4547688388388dd4c7e638 P f2fdd596a8b44bbc848fe40f9f07fbe0: No bootstrap required, opened a new log
I20240502 21:28:50.233587 10138 tablet_bootstrap.cc:492] T 41ee481a3ece47a0905e93c66dbf949d P 5543ddab5b484642b139b7c848932a58: No bootstrap required, opened a new log
I20240502 21:28:50.233912 10138 ts_tablet_manager.cc:1367] T 41ee481a3ece47a0905e93c66dbf949d P 5543ddab5b484642b139b7c848932a58: Time spent bootstrapping tablet: real 0.017s	user 0.007s	sys 0.000s
I20240502 21:28:50.234349 10134 ts_tablet_manager.cc:1367] T de3287296c4547688388388dd4c7e638 P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent bootstrapping tablet: real 0.012s	user 0.010s	sys 0.000s
I20240502 21:28:50.235934 10139 tablet_bootstrap.cc:492] T de3287296c4547688388388dd4c7e638 P 19f162e164854f2da623a4bb8f9e6bf6: No bootstrap required, opened a new log
I20240502 21:28:50.236272 10139 ts_tablet_manager.cc:1367] T de3287296c4547688388388dd4c7e638 P 19f162e164854f2da623a4bb8f9e6bf6: Time spent bootstrapping tablet: real 0.017s	user 0.009s	sys 0.000s
I20240502 21:28:50.244768 10135 tablet_bootstrap.cc:492] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 525a093ea5bf40dc9bbf9dc39895d670: No bootstrap required, opened a new log
I20240502 21:28:50.245062 10135 ts_tablet_manager.cc:1367] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 525a093ea5bf40dc9bbf9dc39895d670: Time spent bootstrapping tablet: real 0.014s	user 0.002s	sys 0.008s
I20240502 21:28:50.245946 10134 raft_consensus.cc:348] T de3287296c4547688388388dd4c7e638 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:50.245906 10139 raft_consensus.cc:348] T de3287296c4547688388388dd4c7e638 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:50.246248 10135 raft_consensus.cc:348] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:50.246438 10139 raft_consensus.cc:374] T de3287296c4547688388388dd4c7e638 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:50.246482 10135 raft_consensus.cc:374] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:50.246552 10134 raft_consensus.cc:374] T de3287296c4547688388388dd4c7e638 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:50.246274 10138 raft_consensus.cc:348] T 41ee481a3ece47a0905e93c66dbf949d P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:50.246660 10135 raft_consensus.cc:729] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Initialized, Role: FOLLOWER
I20240502 21:28:50.246623 10139 raft_consensus.cc:729] T de3287296c4547688388388dd4c7e638 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Initialized, Role: FOLLOWER
I20240502 21:28:50.246701 10138 raft_consensus.cc:374] T 41ee481a3ece47a0905e93c66dbf949d P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:50.246731 10134 raft_consensus.cc:729] T de3287296c4547688388388dd4c7e638 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Initialized, Role: FOLLOWER
I20240502 21:28:50.246922 10138 raft_consensus.cc:729] T 41ee481a3ece47a0905e93c66dbf949d P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Initialized, Role: FOLLOWER
I20240502 21:28:50.246995 10135 consensus_queue.cc:260] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:50.247558 10139 consensus_queue.cc:260] T de3287296c4547688388388dd4c7e638 P 19f162e164854f2da623a4bb8f9e6bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:50.248142 10135 ts_tablet_manager.cc:1398] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 525a093ea5bf40dc9bbf9dc39895d670: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 21:28:50.247570 10134 consensus_queue.cc:260] T de3287296c4547688388388dd4c7e638 P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:50.248301 10138 consensus_queue.cc:260] T 41ee481a3ece47a0905e93c66dbf949d P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:50.250820 10139 ts_tablet_manager.cc:1398] T de3287296c4547688388388dd4c7e638 P 19f162e164854f2da623a4bb8f9e6bf6: Time spent starting tablet: real 0.014s	user 0.014s	sys 0.000s
I20240502 21:28:50.251571 10139 tablet_bootstrap.cc:492] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 19f162e164854f2da623a4bb8f9e6bf6: Bootstrap starting.
I20240502 21:28:50.254180 10139 tablet_bootstrap.cc:654] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 19f162e164854f2da623a4bb8f9e6bf6: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:50.259081 10138 ts_tablet_manager.cc:1398] T 41ee481a3ece47a0905e93c66dbf949d P 5543ddab5b484642b139b7c848932a58: Time spent starting tablet: real 0.022s	user 0.019s	sys 0.000s
I20240502 21:28:50.259565 10139 tablet_bootstrap.cc:492] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 19f162e164854f2da623a4bb8f9e6bf6: No bootstrap required, opened a new log
I20240502 21:28:50.259783 10138 tablet_bootstrap.cc:492] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58: Bootstrap starting.
I20240502 21:28:50.259989 10139 ts_tablet_manager.cc:1367] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 19f162e164854f2da623a4bb8f9e6bf6: Time spent bootstrapping tablet: real 0.009s	user 0.005s	sys 0.000s
I20240502 21:28:50.261596 10138 tablet_bootstrap.cc:654] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:50.262427 10134 ts_tablet_manager.cc:1398] T de3287296c4547688388388dd4c7e638 P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent starting tablet: real 0.028s	user 0.009s	sys 0.012s
I20240502 21:28:50.262966 10134 tablet_bootstrap.cc:492] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap starting.
I20240502 21:28:50.263432 10139 raft_consensus.cc:348] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:50.264279 10139 raft_consensus.cc:374] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:50.264477 10139 raft_consensus.cc:729] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Initialized, Role: FOLLOWER
I20240502 21:28:50.265129 10134 tablet_bootstrap.cc:654] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:50.265408 10139 consensus_queue.cc:260] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 19f162e164854f2da623a4bb8f9e6bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:50.266111 10139 ts_tablet_manager.cc:1398] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 19f162e164854f2da623a4bb8f9e6bf6: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240502 21:28:50.267071 10138 tablet_bootstrap.cc:492] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58: No bootstrap required, opened a new log
I20240502 21:28:50.267334 10138 ts_tablet_manager.cc:1367] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58: Time spent bootstrapping tablet: real 0.008s	user 0.000s	sys 0.006s
I20240502 21:28:50.267686 10134 tablet_bootstrap.cc:492] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0: No bootstrap required, opened a new log
I20240502 21:28:50.267881 10134 ts_tablet_manager.cc:1367] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 21:28:50.268647 10138 raft_consensus.cc:348] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:50.268895 10138 raft_consensus.cc:374] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:50.268801 10134 raft_consensus.cc:348] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:50.269006 10134 raft_consensus.cc:374] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:50.269060 10138 raft_consensus.cc:729] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Initialized, Role: FOLLOWER
I20240502 21:28:50.269153 10134 raft_consensus.cc:729] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Initialized, Role: FOLLOWER
I20240502 21:28:50.269364 10138 consensus_queue.cc:260] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:50.269476 10134 consensus_queue.cc:260] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:50.270099 10134 ts_tablet_manager.cc:1398] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 21:28:50.270216 10138 ts_tablet_manager.cc:1398] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 21:28:50.270606 10138 tablet_bootstrap.cc:492] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58: Bootstrap starting.
I20240502 21:28:50.272652 10138 tablet_bootstrap.cc:654] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:50.274701 10138 tablet_bootstrap.cc:492] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58: No bootstrap required, opened a new log
I20240502 21:28:50.274883 10138 ts_tablet_manager.cc:1367] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 21:28:50.275595 10138 raft_consensus.cc:348] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:50.275802 10138 raft_consensus.cc:374] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:50.275882 10138 raft_consensus.cc:729] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Initialized, Role: FOLLOWER
I20240502 21:28:50.276154 10138 consensus_queue.cc:260] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:50.276723 10138 ts_tablet_manager.cc:1398] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 21:28:50.278096 10146 raft_consensus.cc:482] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:50.278460 10146 raft_consensus.cc:504] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:50.281152 10146 leader_election.cc:288] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:28:50.293011  9378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de3287296c4547688388388dd4c7e638" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" is_pre_election: true
I20240502 21:28:50.293848  9378 raft_consensus.cc:2454] T de3287296c4547688388388dd4c7e638 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 0.
I20240502 21:28:50.294940  9040 leader_election.cc:302] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5543ddab5b484642b139b7c848932a58, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:28:50.295019  9511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de3287296c4547688388388dd4c7e638" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6" is_pre_election: true
I20240502 21:28:50.295460 10146 raft_consensus.cc:2781] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:50.295637 10146 raft_consensus.cc:482] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:50.295737 10146 raft_consensus.cc:3037] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:50.295795  9511 raft_consensus.cc:2454] T de3287296c4547688388388dd4c7e638 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 0.
I20240502 21:28:50.298859 10146 raft_consensus.cc:504] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:50.299775 10146 leader_election.cc:288] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 election: Requested vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:28:50.300851  9378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de3287296c4547688388388dd4c7e638" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
I20240502 21:28:50.300848  9511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de3287296c4547688388388dd4c7e638" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6"
I20240502 21:28:50.301156  9378 raft_consensus.cc:3037] T de3287296c4547688388388dd4c7e638 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:50.301191  9511 raft_consensus.cc:3037] T de3287296c4547688388388dd4c7e638 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:50.304728  9511 raft_consensus.cc:2454] T de3287296c4547688388388dd4c7e638 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 1.
I20240502 21:28:50.304728  9378 raft_consensus.cc:2454] T de3287296c4547688388388dd4c7e638 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 1.
I20240502 21:28:50.305583  9040 leader_election.cc:302] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5543ddab5b484642b139b7c848932a58, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:28:50.306074 10146 raft_consensus.cc:2781] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:50.306689 10146 raft_consensus.cc:686] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Becoming Leader. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Running, Role: LEADER
I20240502 21:28:50.307494 10146 consensus_queue.cc:237] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:50.313275  8962 catalog_manager.cc:5484] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5543ddab5b484642b139b7c848932a58 (127.27.196.65). New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } }
W20240502 21:28:50.319579  9426 tablet.cc:2369] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:28:50.347195 10147 raft_consensus.cc:482] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:50.347476 10147 raft_consensus.cc:504] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:50.349468 10147 leader_election.cc:288] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:28:50.358659  9108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ee481a3ece47a0905e93c66dbf949d" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5543ddab5b484642b139b7c848932a58" is_pre_election: true
I20240502 21:28:50.358789  9242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ee481a3ece47a0905e93c66dbf949d" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670" is_pre_election: true
I20240502 21:28:50.359153  9108 raft_consensus.cc:2454] T 41ee481a3ece47a0905e93c66dbf949d P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 0.
I20240502 21:28:50.359411  9242 raft_consensus.cc:2454] T 41ee481a3ece47a0905e93c66dbf949d P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 0.
I20240502 21:28:50.359767  9310 leader_election.cc:302] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5543ddab5b484642b139b7c848932a58, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:28:50.360255 10147 raft_consensus.cc:2781] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:50.360391 10147 raft_consensus.cc:482] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:50.360486 10147 raft_consensus.cc:3037] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:50.362708 10147 raft_consensus.cc:504] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:50.363610 10147 leader_election.cc:288] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 election: Requested vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:28:50.364130  9242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ee481a3ece47a0905e93c66dbf949d" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670"
I20240502 21:28:50.364194  9108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ee481a3ece47a0905e93c66dbf949d" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5543ddab5b484642b139b7c848932a58"
I20240502 21:28:50.364348  9242 raft_consensus.cc:3037] T 41ee481a3ece47a0905e93c66dbf949d P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:50.364456  9108 raft_consensus.cc:3037] T 41ee481a3ece47a0905e93c66dbf949d P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:50.366679  9242 raft_consensus.cc:2454] T 41ee481a3ece47a0905e93c66dbf949d P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 1.
I20240502 21:28:50.366708  9108 raft_consensus.cc:2454] T 41ee481a3ece47a0905e93c66dbf949d P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 1.
I20240502 21:28:50.367218  9310 leader_election.cc:302] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5543ddab5b484642b139b7c848932a58, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:28:50.367619 10147 raft_consensus.cc:2781] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:50.368093 10147 raft_consensus.cc:686] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Becoming Leader. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Running, Role: LEADER
I20240502 21:28:50.368669 10147 consensus_queue.cc:237] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:50.373710  8962 catalog_manager.cc:5484] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67). New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } }
W20240502 21:28:50.383411  9154 tablet.cc:2369] T 41ee481a3ece47a0905e93c66dbf949d P 5543ddab5b484642b139b7c848932a58: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 21:28:50.404911  9562 tablet.cc:2369] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 19f162e164854f2da623a4bb8f9e6bf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 21:28:50.481623  9290 tablet.cc:2369] T 41ee481a3ece47a0905e93c66dbf949d P 525a093ea5bf40dc9bbf9dc39895d670: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:28:50.500377 10146 raft_consensus.cc:482] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:50.500802 10146 raft_consensus.cc:504] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:50.502163 10146 leader_election.cc:288] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:28:50.502780  9511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9240163cfac446d2a14dbd9fcbbf9ff7" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6" is_pre_election: true
I20240502 21:28:50.503140  9511 raft_consensus.cc:2454] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 0.
I20240502 21:28:50.503732  9038 leader_election.cc:302] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 5543ddab5b484642b139b7c848932a58; no voters: 
I20240502 21:28:50.504158 10146 raft_consensus.cc:2781] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:50.504280 10146 raft_consensus.cc:482] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:50.504402 10146 raft_consensus.cc:3037] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:50.506599 10146 raft_consensus.cc:504] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:50.507452 10146 leader_election.cc:288] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 election: Requested vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:28:50.508455  9511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9240163cfac446d2a14dbd9fcbbf9ff7" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6"
I20240502 21:28:50.508790  9511 raft_consensus.cc:3037] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:50.509546  9242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9240163cfac446d2a14dbd9fcbbf9ff7" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670" is_pre_election: true
I20240502 21:28:50.509814  9242 raft_consensus.cc:2454] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 0.
I20240502 21:28:50.509737  9241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9240163cfac446d2a14dbd9fcbbf9ff7" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670"
I20240502 21:28:50.510098  9241 raft_consensus.cc:3037] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:50.511749  9511 raft_consensus.cc:2454] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 1.
I20240502 21:28:50.512347  9038 leader_election.cc:302] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 5543ddab5b484642b139b7c848932a58; no voters: 
I20240502 21:28:50.512887 10146 raft_consensus.cc:2781] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:50.513159  9241 raft_consensus.cc:2454] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 1.
I20240502 21:28:50.513202 10146 raft_consensus.cc:686] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Becoming Leader. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Running, Role: LEADER
I20240502 21:28:50.513623 10146 consensus_queue.cc:237] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:50.517472  8962 catalog_manager.cc:5484] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5543ddab5b484642b139b7c848932a58 (127.27.196.65). New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } }
I20240502 21:28:50.763255 10147 consensus_queue.cc:1035] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:50.774492 10147 consensus_queue.cc:1035] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:50.837864 10146 consensus_queue.cc:1035] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:50.846017 10146 consensus_queue.cc:1035] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:50.936653 10146 consensus_queue.cc:1035] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:50.944067 10154 consensus_queue.cc:1035] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:28:50.975601 10162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:50.976017 10162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:50.985930 10162 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'

INSERT report
    rows total: 2000
    time total: 321.118 ms
  time per row: 0.160559 ms
I20240502 21:28:51.482900  8963 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54676:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 21:28:51.494531  9357 tablet_service.cc:1467] Processing CreateTablet for tablet 58c7004ceb044f5fa9dec54910046b69 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f52c6c3804274777abfa00bb33ae9b16]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.495218  9357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c7004ceb044f5fa9dec54910046b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.495421  9221 tablet_service.cc:1467] Processing CreateTablet for tablet 07d693b680a54e12871121737e3da744 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f52c6c3804274777abfa00bb33ae9b16]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.495880  9221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d693b680a54e12871121737e3da744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.496742  9491 tablet_service.cc:1467] Processing CreateTablet for tablet 07d693b680a54e12871121737e3da744 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f52c6c3804274777abfa00bb33ae9b16]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.497251  9491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d693b680a54e12871121737e3da744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.494302  9220 tablet_service.cc:1467] Processing CreateTablet for tablet 58c7004ceb044f5fa9dec54910046b69 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f52c6c3804274777abfa00bb33ae9b16]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.500023  9086 tablet_service.cc:1467] Processing CreateTablet for tablet 4d1a31513bb64578a85adf1f622e5bc0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f52c6c3804274777abfa00bb33ae9b16]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.500214  9220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c7004ceb044f5fa9dec54910046b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.500496  9086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d1a31513bb64578a85adf1f622e5bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.502686  9490 tablet_service.cc:1467] Processing CreateTablet for tablet 58c7004ceb044f5fa9dec54910046b69 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f52c6c3804274777abfa00bb33ae9b16]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.503216  9490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c7004ceb044f5fa9dec54910046b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.508634  9219 tablet_service.cc:1467] Processing CreateTablet for tablet 4d1a31513bb64578a85adf1f622e5bc0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f52c6c3804274777abfa00bb33ae9b16]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.509299  9219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d1a31513bb64578a85adf1f622e5bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.509703  9356 tablet_service.cc:1467] Processing CreateTablet for tablet 07d693b680a54e12871121737e3da744 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f52c6c3804274777abfa00bb33ae9b16]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.510285  9356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d693b680a54e12871121737e3da744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.511806 10234 tablet_bootstrap.cc:492] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58: Bootstrap starting.
I20240502 21:28:51.511837  9355 tablet_service.cc:1467] Processing CreateTablet for tablet 4d1a31513bb64578a85adf1f622e5bc0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f52c6c3804274777abfa00bb33ae9b16]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.512346  9355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d1a31513bb64578a85adf1f622e5bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.513173 10233 tablet_bootstrap.cc:492] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6: Bootstrap starting.
I20240502 21:28:51.513872 10234 tablet_bootstrap.cc:654] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.515199 10233 tablet_bootstrap.cc:654] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.529453 10232 tablet_bootstrap.cc:492] T 58c7004ceb044f5fa9dec54910046b69 P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap starting.
I20240502 21:28:51.530726 10235 tablet_bootstrap.cc:492] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap starting.
I20240502 21:28:51.530892 10234 tablet_bootstrap.cc:492] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58: No bootstrap required, opened a new log
I20240502 21:28:51.531157 10234 ts_tablet_manager.cc:1367] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58: Time spent bootstrapping tablet: real 0.019s	user 0.007s	sys 0.000s
I20240502 21:28:51.531776 10232 tablet_bootstrap.cc:654] T 58c7004ceb044f5fa9dec54910046b69 P f2fdd596a8b44bbc848fe40f9f07fbe0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.533370 10235 tablet_bootstrap.cc:654] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.532845 10234 raft_consensus.cc:348] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:51.534250 10234 raft_consensus.cc:374] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.534940 10234 raft_consensus.cc:729] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.535248 10233 tablet_bootstrap.cc:492] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6: No bootstrap required, opened a new log
I20240502 21:28:51.535583 10233 ts_tablet_manager.cc:1367] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6: Time spent bootstrapping tablet: real 0.023s	user 0.001s	sys 0.020s
I20240502 21:28:51.535537 10234 consensus_queue.cc:260] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:51.536481 10234 ts_tablet_manager.cc:1398] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 21:28:51.537710 10233 raft_consensus.cc:348] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:51.538034 10233 raft_consensus.cc:374] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.538198 10233 raft_consensus.cc:729] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.538277 10232 tablet_bootstrap.cc:492] T 58c7004ceb044f5fa9dec54910046b69 P f2fdd596a8b44bbc848fe40f9f07fbe0: No bootstrap required, opened a new log
I20240502 21:28:51.538303 10235 tablet_bootstrap.cc:492] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670: No bootstrap required, opened a new log
I20240502 21:28:51.538455 10232 ts_tablet_manager.cc:1367] T 58c7004ceb044f5fa9dec54910046b69 P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.004s
I20240502 21:28:51.538463 10235 ts_tablet_manager.cc:1367] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.003s
I20240502 21:28:51.539458 10232 raft_consensus.cc:348] T 58c7004ceb044f5fa9dec54910046b69 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:51.539682 10232 raft_consensus.cc:374] T 58c7004ceb044f5fa9dec54910046b69 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.539777 10232 raft_consensus.cc:729] T 58c7004ceb044f5fa9dec54910046b69 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.539476 10233 consensus_queue.cc:260] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:51.540043 10232 consensus_queue.cc:260] T 58c7004ceb044f5fa9dec54910046b69 P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:51.540285 10235 raft_consensus.cc:348] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:51.540529 10235 raft_consensus.cc:374] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.540625 10235 raft_consensus.cc:729] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.540678 10232 ts_tablet_manager.cc:1398] T 58c7004ceb044f5fa9dec54910046b69 P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 21:28:51.540998 10232 tablet_bootstrap.cc:492] T 4d1a31513bb64578a85adf1f622e5bc0 P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap starting.
I20240502 21:28:51.540931 10235 consensus_queue.cc:260] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:51.542598 10235 ts_tablet_manager.cc:1398] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 21:28:51.543393 10235 tablet_bootstrap.cc:492] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap starting.
I20240502 21:28:51.543900 10232 tablet_bootstrap.cc:654] T 4d1a31513bb64578a85adf1f622e5bc0 P f2fdd596a8b44bbc848fe40f9f07fbe0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.546332 10235 tablet_bootstrap.cc:654] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.548274 10233 ts_tablet_manager.cc:1398] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6: Time spent starting tablet: real 0.012s	user 0.003s	sys 0.000s
I20240502 21:28:51.548826 10233 tablet_bootstrap.cc:492] T 58c7004ceb044f5fa9dec54910046b69 P 19f162e164854f2da623a4bb8f9e6bf6: Bootstrap starting.
I20240502 21:28:51.551734 10233 tablet_bootstrap.cc:654] T 58c7004ceb044f5fa9dec54910046b69 P 19f162e164854f2da623a4bb8f9e6bf6: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.553056 10232 tablet_bootstrap.cc:492] T 4d1a31513bb64578a85adf1f622e5bc0 P f2fdd596a8b44bbc848fe40f9f07fbe0: No bootstrap required, opened a new log
I20240502 21:28:51.553467 10232 ts_tablet_manager.cc:1367] T 4d1a31513bb64578a85adf1f622e5bc0 P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent bootstrapping tablet: real 0.012s	user 0.008s	sys 0.003s
I20240502 21:28:51.554785 10232 raft_consensus.cc:348] T 4d1a31513bb64578a85adf1f622e5bc0 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:51.555238 10232 raft_consensus.cc:374] T 4d1a31513bb64578a85adf1f622e5bc0 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.555398 10232 raft_consensus.cc:729] T 4d1a31513bb64578a85adf1f622e5bc0 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.555761 10232 consensus_queue.cc:260] T 4d1a31513bb64578a85adf1f622e5bc0 P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:51.555945 10235 tablet_bootstrap.cc:492] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670: No bootstrap required, opened a new log
I20240502 21:28:51.556139 10235 ts_tablet_manager.cc:1367] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670: Time spent bootstrapping tablet: real 0.013s	user 0.003s	sys 0.006s
I20240502 21:28:51.556526 10232 ts_tablet_manager.cc:1398] T 4d1a31513bb64578a85adf1f622e5bc0 P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.001s
I20240502 21:28:51.556979 10232 tablet_bootstrap.cc:492] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap starting.
I20240502 21:28:51.556946 10235 raft_consensus.cc:348] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:51.557266 10235 raft_consensus.cc:374] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.557404 10235 raft_consensus.cc:729] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.557473 10233 tablet_bootstrap.cc:492] T 58c7004ceb044f5fa9dec54910046b69 P 19f162e164854f2da623a4bb8f9e6bf6: No bootstrap required, opened a new log
I20240502 21:28:51.557713 10233 ts_tablet_manager.cc:1367] T 58c7004ceb044f5fa9dec54910046b69 P 19f162e164854f2da623a4bb8f9e6bf6: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.005s
I20240502 21:28:51.557757 10235 consensus_queue.cc:260] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:51.558926 10232 tablet_bootstrap.cc:654] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.558687 10233 raft_consensus.cc:348] T 58c7004ceb044f5fa9dec54910046b69 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:51.559975 10233 raft_consensus.cc:374] T 58c7004ceb044f5fa9dec54910046b69 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.560125 10233 raft_consensus.cc:729] T 58c7004ceb044f5fa9dec54910046b69 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.561232 10235 ts_tablet_manager.cc:1398] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 21:28:51.561044 10233 consensus_queue.cc:260] T 58c7004ceb044f5fa9dec54910046b69 P 19f162e164854f2da623a4bb8f9e6bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:51.562057 10235 tablet_bootstrap.cc:492] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap starting.
I20240502 21:28:51.562602 10233 ts_tablet_manager.cc:1398] T 58c7004ceb044f5fa9dec54910046b69 P 19f162e164854f2da623a4bb8f9e6bf6: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.002s
I20240502 21:28:51.565249 10232 tablet_bootstrap.cc:492] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0: No bootstrap required, opened a new log
I20240502 21:28:51.565479 10232 ts_tablet_manager.cc:1367] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent bootstrapping tablet: real 0.009s	user 0.007s	sys 0.000s
I20240502 21:28:51.565779 10235 tablet_bootstrap.cc:654] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.566643 10232 raft_consensus.cc:348] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:51.566944 10232 raft_consensus.cc:374] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.567138 10232 raft_consensus.cc:729] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.567476 10232 consensus_queue.cc:260] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:51.568082 10232 ts_tablet_manager.cc:1398] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 21:28:51.569185 10235 tablet_bootstrap.cc:492] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670: No bootstrap required, opened a new log
I20240502 21:28:51.569360 10235 ts_tablet_manager.cc:1367] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670: Time spent bootstrapping tablet: real 0.008s	user 0.004s	sys 0.000s
I20240502 21:28:51.570205 10235 raft_consensus.cc:348] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:51.570463 10235 raft_consensus.cc:374] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.570569 10235 raft_consensus.cc:729] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.570887 10235 consensus_queue.cc:260] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:51.571563 10235 ts_tablet_manager.cc:1398] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 21:28:51.576138 10241 raft_consensus.cc:482] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:51.576370 10241 raft_consensus.cc:504] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:51.578316 10241 leader_election.cc:288] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:28:51.588554  9378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d693b680a54e12871121737e3da744" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" is_pre_election: true
I20240502 21:28:51.588881  9378 raft_consensus.cc:2454] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 0.
I20240502 21:28:51.588938  9511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d693b680a54e12871121737e3da744" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6" is_pre_election: true
I20240502 21:28:51.589265  9511 raft_consensus.cc:2454] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 0.
I20240502 21:28:51.589557  9174 leader_election.cc:302] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525a093ea5bf40dc9bbf9dc39895d670, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:28:51.590080 10241 raft_consensus.cc:2781] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:51.590210 10241 raft_consensus.cc:482] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:51.590566 10241 raft_consensus.cc:3037] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.593559 10241 raft_consensus.cc:504] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:51.594527 10241 leader_election.cc:288] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 election: Requested vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:28:51.595363  9511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d693b680a54e12871121737e3da744" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6"
I20240502 21:28:51.595345  9378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d693b680a54e12871121737e3da744" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
I20240502 21:28:51.595702  9511 raft_consensus.cc:3037] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.595731  9378 raft_consensus.cc:3037] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.599081  9378 raft_consensus.cc:2454] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 1.
I20240502 21:28:51.599156  9511 raft_consensus.cc:2454] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 1.
I20240502 21:28:51.599988  9172 leader_election.cc:302] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 525a093ea5bf40dc9bbf9dc39895d670, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:28:51.600502 10241 raft_consensus.cc:2781] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:51.601168 10241 raft_consensus.cc:686] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Becoming Leader. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Running, Role: LEADER
I20240502 21:28:51.602181 10241 consensus_queue.cc:237] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:51.606241  8958 catalog_manager.cc:5484] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } }
I20240502 21:28:51.688657 10241 raft_consensus.cc:482] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:51.688977 10241 raft_consensus.cc:504] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:51.689798 10241 leader_election.cc:288] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
I20240502 21:28:51.690590  9378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c7004ceb044f5fa9dec54910046b69" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" is_pre_election: true
I20240502 21:28:51.690598  9511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c7004ceb044f5fa9dec54910046b69" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6" is_pre_election: true
I20240502 21:28:51.690869  9378 raft_consensus.cc:2454] T 58c7004ceb044f5fa9dec54910046b69 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 0.
I20240502 21:28:51.690898  9511 raft_consensus.cc:2454] T 58c7004ceb044f5fa9dec54910046b69 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 0.
I20240502 21:28:51.691411  9172 leader_election.cc:302] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 525a093ea5bf40dc9bbf9dc39895d670; no voters: 
I20240502 21:28:51.691773 10241 raft_consensus.cc:2781] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:51.691888 10241 raft_consensus.cc:482] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:51.691980 10241 raft_consensus.cc:3037] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.694110 10241 raft_consensus.cc:504] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:51.695212 10241 leader_election.cc:288] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 election: Requested vote from peers f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
I20240502 21:28:51.695827  9378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c7004ceb044f5fa9dec54910046b69" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
I20240502 21:28:51.696070  9378 raft_consensus.cc:3037] T 58c7004ceb044f5fa9dec54910046b69 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.696022  9511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c7004ceb044f5fa9dec54910046b69" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6"
I20240502 21:28:51.696398  9511 raft_consensus.cc:3037] T 58c7004ceb044f5fa9dec54910046b69 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.698297  9378 raft_consensus.cc:2454] T 58c7004ceb044f5fa9dec54910046b69 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 1.
I20240502 21:28:51.698798  9174 leader_election.cc:302] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525a093ea5bf40dc9bbf9dc39895d670, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:28:51.698845  9511 raft_consensus.cc:2454] T 58c7004ceb044f5fa9dec54910046b69 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 1.
I20240502 21:28:51.699157 10241 raft_consensus.cc:2781] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:51.699360 10241 raft_consensus.cc:686] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Becoming Leader. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Running, Role: LEADER
I20240502 21:28:51.699713 10241 consensus_queue.cc:237] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:51.703022  8958 catalog_manager.cc:5484] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } }
I20240502 21:28:51.737907 10241 raft_consensus.cc:482] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:51.738205 10241 raft_consensus.cc:504] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:51.739575 10241 leader_election.cc:288] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:28:51.740432  9378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1a31513bb64578a85adf1f622e5bc0" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" is_pre_election: true
I20240502 21:28:51.740770  9378 raft_consensus.cc:2454] T 4d1a31513bb64578a85adf1f622e5bc0 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 0.
I20240502 21:28:51.741354  9174 leader_election.cc:302] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525a093ea5bf40dc9bbf9dc39895d670, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:28:51.741744 10241 raft_consensus.cc:2781] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:51.741923 10241 raft_consensus.cc:482] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:51.742100 10241 raft_consensus.cc:3037] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.745157 10241 raft_consensus.cc:504] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:51.746870 10241 leader_election.cc:288] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 election: Requested vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:28:51.747720  9378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1a31513bb64578a85adf1f622e5bc0" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
I20240502 21:28:51.748214  9378 raft_consensus.cc:3037] T 4d1a31513bb64578a85adf1f622e5bc0 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.749413  9108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1a31513bb64578a85adf1f622e5bc0" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5543ddab5b484642b139b7c848932a58" is_pre_election: true
I20240502 21:28:51.749486  9107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1a31513bb64578a85adf1f622e5bc0" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5543ddab5b484642b139b7c848932a58"
I20240502 21:28:51.749758  9108 raft_consensus.cc:2454] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 0.
I20240502 21:28:51.751026  9378 raft_consensus.cc:2454] T 4d1a31513bb64578a85adf1f622e5bc0 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 1.
I20240502 21:28:51.751710  9174 leader_election.cc:302] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525a093ea5bf40dc9bbf9dc39895d670, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 5543ddab5b484642b139b7c848932a58
I20240502 21:28:51.752120 10241 raft_consensus.cc:2781] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:51.752274 10241 raft_consensus.cc:686] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Becoming Leader. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Running, Role: LEADER
I20240502 21:28:51.752573 10241 consensus_queue.cc:237] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:51.755730  8958 catalog_manager.cc:5484] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } }
I20240502 21:28:52.045924 10241 consensus_queue.cc:1035] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:52.052865 10241 consensus_queue.cc:1035] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:52.103444 10241 consensus_queue.cc:1035] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:52.110642 10262 consensus_queue.cc:1035] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:52.137979 10262 consensus_queue.cc:1035] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:52.144747  9107 raft_consensus.cc:3037] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:52.149012 10241 consensus_queue.cc:1035] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:28:52.212587 10255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:52.213035 10255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:52.224190 10255 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
W20240502 21:28:52.451421  9172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.68:40931, user_credentials={real_user=slave}} blocked reactor thread for 50135us
W20240502 21:28:52.453114  9174 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.67:43203, user_credentials={real_user=slave}} blocked reactor thread for 52738.4us

INSERT report
    rows total: 2000
    time total: 209.565 ms
  time per row: 0.104782 ms
I20240502 21:28:52.621476  8963 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54676:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 21:28:52.633137  9490 tablet_service.cc:1467] Processing CreateTablet for tablet 51936fb1f678458a8795f8246590addb (DEFAULT_TABLE table=rebalance_test_table_2 [id=c0a0ef39a2a246449eb582cc1d993484]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:52.633695  9490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51936fb1f678458a8795f8246590addb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:52.634191  9357 tablet_service.cc:1467] Processing CreateTablet for tablet aeee59c5512d448d8f986f959dce1cc8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c0a0ef39a2a246449eb582cc1d993484]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:52.634631  9357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeee59c5512d448d8f986f959dce1cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:52.635665  9220 tablet_service.cc:1467] Processing CreateTablet for tablet 51936fb1f678458a8795f8246590addb (DEFAULT_TABLE table=rebalance_test_table_2 [id=c0a0ef39a2a246449eb582cc1d993484]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:52.635694  9219 tablet_service.cc:1467] Processing CreateTablet for tablet cdf72d25f0f847e5998acbc53979e64d (DEFAULT_TABLE table=rebalance_test_table_2 [id=c0a0ef39a2a246449eb582cc1d993484]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:52.636173  9220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51936fb1f678458a8795f8246590addb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:52.636425  9219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdf72d25f0f847e5998acbc53979e64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:52.640609  9087 tablet_service.cc:1467] Processing CreateTablet for tablet aeee59c5512d448d8f986f959dce1cc8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c0a0ef39a2a246449eb582cc1d993484]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:52.640702  9491 tablet_service.cc:1467] Processing CreateTablet for tablet aeee59c5512d448d8f986f959dce1cc8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c0a0ef39a2a246449eb582cc1d993484]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:52.641158  9491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeee59c5512d448d8f986f959dce1cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:52.641158  9087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeee59c5512d448d8f986f959dce1cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:52.642725  9085 tablet_service.cc:1467] Processing CreateTablet for tablet cdf72d25f0f847e5998acbc53979e64d (DEFAULT_TABLE table=rebalance_test_table_2 [id=c0a0ef39a2a246449eb582cc1d993484]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:52.643206  9085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdf72d25f0f847e5998acbc53979e64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:52.640635  9086 tablet_service.cc:1467] Processing CreateTablet for tablet 51936fb1f678458a8795f8246590addb (DEFAULT_TABLE table=rebalance_test_table_2 [id=c0a0ef39a2a246449eb582cc1d993484]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:52.647773  9086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51936fb1f678458a8795f8246590addb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:52.647778  9489 tablet_service.cc:1467] Processing CreateTablet for tablet cdf72d25f0f847e5998acbc53979e64d (DEFAULT_TABLE table=rebalance_test_table_2 [id=c0a0ef39a2a246449eb582cc1d993484]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:52.648223  9489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdf72d25f0f847e5998acbc53979e64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:52.658712 10317 tablet_bootstrap.cc:492] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap starting.
I20240502 21:28:52.660773 10317 tablet_bootstrap.cc:654] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:52.664884 10320 tablet_bootstrap.cc:492] T cdf72d25f0f847e5998acbc53979e64d P 19f162e164854f2da623a4bb8f9e6bf6: Bootstrap starting.
I20240502 21:28:52.666874 10317 tablet_bootstrap.cc:492] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0: No bootstrap required, opened a new log
I20240502 21:28:52.667447 10319 tablet_bootstrap.cc:492] T aeee59c5512d448d8f986f959dce1cc8 P 5543ddab5b484642b139b7c848932a58: Bootstrap starting.
I20240502 21:28:52.667886 10317 ts_tablet_manager.cc:1367] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.000s
I20240502 21:28:52.668655 10320 tablet_bootstrap.cc:654] T cdf72d25f0f847e5998acbc53979e64d P 19f162e164854f2da623a4bb8f9e6bf6: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:52.668831 10317 raft_consensus.cc:348] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:52.669035 10317 raft_consensus.cc:374] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:52.669320 10319 tablet_bootstrap.cc:654] T aeee59c5512d448d8f986f959dce1cc8 P 5543ddab5b484642b139b7c848932a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:52.670646 10317 raft_consensus.cc:729] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Initialized, Role: FOLLOWER
I20240502 21:28:52.671182 10320 tablet_bootstrap.cc:492] T cdf72d25f0f847e5998acbc53979e64d P 19f162e164854f2da623a4bb8f9e6bf6: No bootstrap required, opened a new log
I20240502 21:28:52.671051 10317 consensus_queue.cc:260] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:52.671422 10320 ts_tablet_manager.cc:1367] T cdf72d25f0f847e5998acbc53979e64d P 19f162e164854f2da623a4bb8f9e6bf6: Time spent bootstrapping tablet: real 0.007s	user 0.004s	sys 0.001s
I20240502 21:28:52.672186 10319 tablet_bootstrap.cc:492] T aeee59c5512d448d8f986f959dce1cc8 P 5543ddab5b484642b139b7c848932a58: No bootstrap required, opened a new log
I20240502 21:28:52.672389 10319 ts_tablet_manager.cc:1367] T aeee59c5512d448d8f986f959dce1cc8 P 5543ddab5b484642b139b7c848932a58: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.004s
I20240502 21:28:52.672567 10317 ts_tablet_manager.cc:1398] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 21:28:52.673089 10320 raft_consensus.cc:348] T cdf72d25f0f847e5998acbc53979e64d P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:52.673275 10320 raft_consensus.cc:374] T cdf72d25f0f847e5998acbc53979e64d P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:52.673368 10320 raft_consensus.cc:729] T cdf72d25f0f847e5998acbc53979e64d P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Initialized, Role: FOLLOWER
I20240502 21:28:52.673280 10319 raft_consensus.cc:348] T aeee59c5512d448d8f986f959dce1cc8 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:52.673583 10319 raft_consensus.cc:374] T aeee59c5512d448d8f986f959dce1cc8 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:52.673606 10320 consensus_queue.cc:260] T cdf72d25f0f847e5998acbc53979e64d P 19f162e164854f2da623a4bb8f9e6bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:52.673803 10319 raft_consensus.cc:729] T aeee59c5512d448d8f986f959dce1cc8 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Initialized, Role: FOLLOWER
I20240502 21:28:52.674149 10318 tablet_bootstrap.cc:492] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap starting.
I20240502 21:28:52.674142 10319 consensus_queue.cc:260] T aeee59c5512d448d8f986f959dce1cc8 P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:52.675493 10319 ts_tablet_manager.cc:1398] T aeee59c5512d448d8f986f959dce1cc8 P 5543ddab5b484642b139b7c848932a58: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 21:28:52.675865 10319 tablet_bootstrap.cc:492] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58: Bootstrap starting.
I20240502 21:28:52.676136 10318 tablet_bootstrap.cc:654] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:52.677569 10320 ts_tablet_manager.cc:1398] T cdf72d25f0f847e5998acbc53979e64d P 19f162e164854f2da623a4bb8f9e6bf6: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.002s
I20240502 21:28:52.677918 10320 tablet_bootstrap.cc:492] T aeee59c5512d448d8f986f959dce1cc8 P 19f162e164854f2da623a4bb8f9e6bf6: Bootstrap starting.
I20240502 21:28:52.678592 10319 tablet_bootstrap.cc:654] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:52.679592 10320 tablet_bootstrap.cc:654] T aeee59c5512d448d8f986f959dce1cc8 P 19f162e164854f2da623a4bb8f9e6bf6: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:52.682857 10318 tablet_bootstrap.cc:492] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670: No bootstrap required, opened a new log
I20240502 21:28:52.683176 10318 ts_tablet_manager.cc:1367] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.000s
I20240502 21:28:52.684211 10318 raft_consensus.cc:348] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:52.684456 10318 raft_consensus.cc:374] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:52.684510 10320 tablet_bootstrap.cc:492] T aeee59c5512d448d8f986f959dce1cc8 P 19f162e164854f2da623a4bb8f9e6bf6: No bootstrap required, opened a new log
I20240502 21:28:52.684581 10318 raft_consensus.cc:729] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Initialized, Role: FOLLOWER
I20240502 21:28:52.684686 10320 ts_tablet_manager.cc:1367] T aeee59c5512d448d8f986f959dce1cc8 P 19f162e164854f2da623a4bb8f9e6bf6: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.003s
I20240502 21:28:52.684882 10318 consensus_queue.cc:260] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:52.685379 10319 tablet_bootstrap.cc:492] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58: No bootstrap required, opened a new log
I20240502 21:28:52.685624 10319 ts_tablet_manager.cc:1367] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58: Time spent bootstrapping tablet: real 0.010s	user 0.006s	sys 0.001s
I20240502 21:28:52.685664 10318 ts_tablet_manager.cc:1398] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:28:52.685658 10320 raft_consensus.cc:348] T aeee59c5512d448d8f986f959dce1cc8 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:52.685927 10320 raft_consensus.cc:374] T aeee59c5512d448d8f986f959dce1cc8 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:52.686065 10320 raft_consensus.cc:729] T aeee59c5512d448d8f986f959dce1cc8 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Initialized, Role: FOLLOWER
I20240502 21:28:52.686136 10318 tablet_bootstrap.cc:492] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap starting.
I20240502 21:28:52.686409 10320 consensus_queue.cc:260] T aeee59c5512d448d8f986f959dce1cc8 P 19f162e164854f2da623a4bb8f9e6bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:52.686511 10319 raft_consensus.cc:348] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:52.686739 10319 raft_consensus.cc:374] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:52.686818 10319 raft_consensus.cc:729] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Initialized, Role: FOLLOWER
I20240502 21:28:52.687073 10319 consensus_queue.cc:260] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:52.687705 10319 ts_tablet_manager.cc:1398] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 21:28:52.688032 10319 tablet_bootstrap.cc:492] T 51936fb1f678458a8795f8246590addb P 5543ddab5b484642b139b7c848932a58: Bootstrap starting.
I20240502 21:28:52.688300 10318 tablet_bootstrap.cc:654] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:52.689895 10319 tablet_bootstrap.cc:654] T 51936fb1f678458a8795f8246590addb P 5543ddab5b484642b139b7c848932a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:52.690848 10320 ts_tablet_manager.cc:1398] T aeee59c5512d448d8f986f959dce1cc8 P 19f162e164854f2da623a4bb8f9e6bf6: Time spent starting tablet: real 0.006s	user 0.001s	sys 0.001s
I20240502 21:28:52.691263 10320 tablet_bootstrap.cc:492] T 51936fb1f678458a8795f8246590addb P 19f162e164854f2da623a4bb8f9e6bf6: Bootstrap starting.
I20240502 21:28:52.693395 10320 tablet_bootstrap.cc:654] T 51936fb1f678458a8795f8246590addb P 19f162e164854f2da623a4bb8f9e6bf6: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:52.693890 10318 tablet_bootstrap.cc:492] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670: No bootstrap required, opened a new log
I20240502 21:28:52.694139 10319 tablet_bootstrap.cc:492] T 51936fb1f678458a8795f8246590addb P 5543ddab5b484642b139b7c848932a58: No bootstrap required, opened a new log
I20240502 21:28:52.694150 10318 ts_tablet_manager.cc:1367] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670: Time spent bootstrapping tablet: real 0.008s	user 0.000s	sys 0.004s
I20240502 21:28:52.694352 10319 ts_tablet_manager.cc:1367] T 51936fb1f678458a8795f8246590addb P 5543ddab5b484642b139b7c848932a58: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.001s
I20240502 21:28:52.695050 10318 raft_consensus.cc:348] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:52.695315 10318 raft_consensus.cc:374] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:52.695413 10318 raft_consensus.cc:729] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Initialized, Role: FOLLOWER
I20240502 21:28:52.695705 10318 consensus_queue.cc:260] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:52.696540 10318 ts_tablet_manager.cc:1398] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:28:52.697607 10319 raft_consensus.cc:348] T 51936fb1f678458a8795f8246590addb P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:52.698030 10319 raft_consensus.cc:374] T 51936fb1f678458a8795f8246590addb P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:52.698205 10319 raft_consensus.cc:729] T 51936fb1f678458a8795f8246590addb P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Initialized, Role: FOLLOWER
I20240502 21:28:52.698840 10319 consensus_queue.cc:260] T 51936fb1f678458a8795f8246590addb P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:52.699676 10319 ts_tablet_manager.cc:1398] T 51936fb1f678458a8795f8246590addb P 5543ddab5b484642b139b7c848932a58: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 21:28:52.700538 10320 tablet_bootstrap.cc:492] T 51936fb1f678458a8795f8246590addb P 19f162e164854f2da623a4bb8f9e6bf6: No bootstrap required, opened a new log
I20240502 21:28:52.700726 10320 ts_tablet_manager.cc:1367] T 51936fb1f678458a8795f8246590addb P 19f162e164854f2da623a4bb8f9e6bf6: Time spent bootstrapping tablet: real 0.010s	user 0.003s	sys 0.004s
I20240502 21:28:52.701647 10320 raft_consensus.cc:348] T 51936fb1f678458a8795f8246590addb P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:52.701841 10320 raft_consensus.cc:374] T 51936fb1f678458a8795f8246590addb P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:52.701982 10320 raft_consensus.cc:729] T 51936fb1f678458a8795f8246590addb P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Initialized, Role: FOLLOWER
I20240502 21:28:52.702253 10320 consensus_queue.cc:260] T 51936fb1f678458a8795f8246590addb P 19f162e164854f2da623a4bb8f9e6bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:52.702798 10320 ts_tablet_manager.cc:1398] T 51936fb1f678458a8795f8246590addb P 19f162e164854f2da623a4bb8f9e6bf6: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 21:28:52.780306 10262 raft_consensus.cc:482] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:52.780560 10262 raft_consensus.cc:504] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:52.781874 10262 leader_election.cc:288] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:28:52.782879  9510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51936fb1f678458a8795f8246590addb" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6" is_pre_election: true
I20240502 21:28:52.782945  9107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51936fb1f678458a8795f8246590addb" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5543ddab5b484642b139b7c848932a58" is_pre_election: true
I20240502 21:28:52.783277  9510 raft_consensus.cc:2454] T 51936fb1f678458a8795f8246590addb P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 0.
I20240502 21:28:52.783283  9107 raft_consensus.cc:2454] T 51936fb1f678458a8795f8246590addb P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 0.
I20240502 21:28:52.784032  9172 leader_election.cc:302] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525a093ea5bf40dc9bbf9dc39895d670, 5543ddab5b484642b139b7c848932a58; no voters: 
I20240502 21:28:52.784538 10262 raft_consensus.cc:2781] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:52.784703 10262 raft_consensus.cc:482] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:52.784821 10262 raft_consensus.cc:3037] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:52.787849 10262 raft_consensus.cc:504] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:52.788596 10262 leader_election.cc:288] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 election: Requested vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:28:52.789660  9107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51936fb1f678458a8795f8246590addb" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5543ddab5b484642b139b7c848932a58"
I20240502 21:28:52.789666  9510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51936fb1f678458a8795f8246590addb" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6"
I20240502 21:28:52.790016  9107 raft_consensus.cc:3037] T 51936fb1f678458a8795f8246590addb P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:52.790019  9510 raft_consensus.cc:3037] T 51936fb1f678458a8795f8246590addb P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:52.793388  9510 raft_consensus.cc:2454] T 51936fb1f678458a8795f8246590addb P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 1.
I20240502 21:28:52.793388  9107 raft_consensus.cc:2454] T 51936fb1f678458a8795f8246590addb P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 1.
I20240502 21:28:52.794377  9172 leader_election.cc:302] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 525a093ea5bf40dc9bbf9dc39895d670; no voters: 
I20240502 21:28:52.794981 10262 raft_consensus.cc:2781] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:52.795214 10262 raft_consensus.cc:686] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Becoming Leader. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Running, Role: LEADER
I20240502 21:28:52.795856 10262 consensus_queue.cc:237] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:52.799818  8960 catalog_manager.cc:5484] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } }
I20240502 21:28:52.891265 10146 raft_consensus.cc:482] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:52.891662 10146 raft_consensus.cc:504] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:52.892566 10146 leader_election.cc:288] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:28:52.893529  9241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf72d25f0f847e5998acbc53979e64d" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670" is_pre_election: true
I20240502 21:28:52.893545  9510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf72d25f0f847e5998acbc53979e64d" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6" is_pre_election: true
I20240502 21:28:52.893864  9510 raft_consensus.cc:2454] T cdf72d25f0f847e5998acbc53979e64d P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 0.
I20240502 21:28:52.893864  9241 raft_consensus.cc:2454] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 0.
I20240502 21:28:52.894405  9038 leader_election.cc:302] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 5543ddab5b484642b139b7c848932a58; no voters: 
I20240502 21:28:52.894807 10146 raft_consensus.cc:2781] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:52.894932 10146 raft_consensus.cc:482] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:52.895036 10146 raft_consensus.cc:3037] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:52.897159 10146 raft_consensus.cc:504] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:52.897903 10146 leader_election.cc:288] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 election: Requested vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:28:52.898825  9510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf72d25f0f847e5998acbc53979e64d" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6"
I20240502 21:28:52.898849  9241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf72d25f0f847e5998acbc53979e64d" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670"
I20240502 21:28:52.899184  9241 raft_consensus.cc:3037] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:52.899282  9510 raft_consensus.cc:3037] T cdf72d25f0f847e5998acbc53979e64d P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:52.902067  9241 raft_consensus.cc:2454] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 1.
I20240502 21:28:52.902484  9510 raft_consensus.cc:2454] T cdf72d25f0f847e5998acbc53979e64d P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 1.
I20240502 21:28:52.902698  9041 leader_election.cc:302] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525a093ea5bf40dc9bbf9dc39895d670, 5543ddab5b484642b139b7c848932a58; no voters: 
I20240502 21:28:52.903155 10146 raft_consensus.cc:2781] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:52.903993 10146 raft_consensus.cc:686] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Becoming Leader. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Running, Role: LEADER
I20240502 21:28:52.904542 10146 consensus_queue.cc:237] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:52.908560  8960 catalog_manager.cc:5484] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5543ddab5b484642b139b7c848932a58 (127.27.196.65). New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } }
I20240502 21:28:52.932245 10147 raft_consensus.cc:482] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:52.932489 10147 raft_consensus.cc:504] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:52.933240 10147 leader_election.cc:288] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:28:52.934684  9107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeee59c5512d448d8f986f959dce1cc8" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5543ddab5b484642b139b7c848932a58" is_pre_election: true
I20240502 21:28:52.935003  9107 raft_consensus.cc:2454] T aeee59c5512d448d8f986f959dce1cc8 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 0.
I20240502 21:28:52.935631  9310 leader_election.cc:302] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5543ddab5b484642b139b7c848932a58, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:28:52.936014 10147 raft_consensus.cc:2781] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:52.936172 10147 raft_consensus.cc:482] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:52.936304 10147 raft_consensus.cc:3037] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:52.938958 10147 raft_consensus.cc:504] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:52.939783 10147 leader_election.cc:288] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 election: Requested vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:28:52.940480  9107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeee59c5512d448d8f986f959dce1cc8" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5543ddab5b484642b139b7c848932a58"
I20240502 21:28:52.940773  9107 raft_consensus.cc:3037] T aeee59c5512d448d8f986f959dce1cc8 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:52.942322  9510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeee59c5512d448d8f986f959dce1cc8" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6" is_pre_election: true
I20240502 21:28:52.942386  9511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeee59c5512d448d8f986f959dce1cc8" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6"
I20240502 21:28:52.942622  9510 raft_consensus.cc:2454] T aeee59c5512d448d8f986f959dce1cc8 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 0.
I20240502 21:28:52.943609  9107 raft_consensus.cc:2454] T aeee59c5512d448d8f986f959dce1cc8 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 1.
I20240502 21:28:52.944025  9310 leader_election.cc:302] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5543ddab5b484642b139b7c848932a58, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 19f162e164854f2da623a4bb8f9e6bf6
I20240502 21:28:52.944372 10147 raft_consensus.cc:2781] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:52.944845 10147 raft_consensus.cc:686] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Becoming Leader. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Running, Role: LEADER
I20240502 21:28:52.945200 10147 consensus_queue.cc:237] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:52.948709  8960 catalog_manager.cc:5484] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67). New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } }
I20240502 21:28:53.247087 10303 consensus_queue.cc:1035] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:53.262077 10303 consensus_queue.cc:1035] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:53.317585 10146 consensus_queue.cc:1035] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:53.323052 10146 consensus_queue.cc:1035] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:53.346132  9511 raft_consensus.cc:3037] T aeee59c5512d448d8f986f959dce1cc8 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:53.349663 10147 consensus_queue.cc:1035] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:53.359146 10147 consensus_queue.cc:1035] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:28:53.400632 10339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:53.401135 10339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:53.414204 10339 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'

INSERT report
    rows total: 2000
    time total: 276.069 ms
  time per row: 0.138034 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
     40      10276 libcrypto.so.1.1
-----------------------------------------------------

I20240502 21:28:54.856439  8953 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54676:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 21:28:54.867360  9219 tablet_service.cc:1467] Processing CreateTablet for tablet 4674e376bf7440cda619784491685b44 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a237d7bda970485480e98020c6ac738f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.867501  9357 tablet_service.cc:1467] Processing CreateTablet for tablet 4674e376bf7440cda619784491685b44 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a237d7bda970485480e98020c6ac738f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.867942  9357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4674e376bf7440cda619784491685b44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.867944  9219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4674e376bf7440cda619784491685b44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.868904  9220 tablet_service.cc:1467] Processing CreateTablet for tablet 1497b781f36c43b39d22a05bb524c59d (DEFAULT_TABLE table=rebalance_test_table_3 [id=a237d7bda970485480e98020c6ac738f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.869405  9220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1497b781f36c43b39d22a05bb524c59d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.869642  9490 tablet_service.cc:1467] Processing CreateTablet for tablet 8b4486f03fd9474aa0bb2cb4c9149437 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a237d7bda970485480e98020c6ac738f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.870114  9490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b4486f03fd9474aa0bb2cb4c9149437. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.875744  9489 tablet_service.cc:1467] Processing CreateTablet for tablet 1497b781f36c43b39d22a05bb524c59d (DEFAULT_TABLE table=rebalance_test_table_3 [id=a237d7bda970485480e98020c6ac738f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.875744  9087 tablet_service.cc:1467] Processing CreateTablet for tablet 4674e376bf7440cda619784491685b44 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a237d7bda970485480e98020c6ac738f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.875769  9086 tablet_service.cc:1467] Processing CreateTablet for tablet 8b4486f03fd9474aa0bb2cb4c9149437 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a237d7bda970485480e98020c6ac738f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.876241  9489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1497b781f36c43b39d22a05bb524c59d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.876312  9087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4674e376bf7440cda619784491685b44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.877660 10415 tablet_bootstrap.cc:492] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap starting.
I20240502 21:28:54.879424  9086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b4486f03fd9474aa0bb2cb4c9149437. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.879719 10414 tablet_bootstrap.cc:492] T 4674e376bf7440cda619784491685b44 P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap starting.
I20240502 21:28:54.879576 10415 tablet_bootstrap.cc:654] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.880431  9355 tablet_service.cc:1467] Processing CreateTablet for tablet 8b4486f03fd9474aa0bb2cb4c9149437 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a237d7bda970485480e98020c6ac738f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.880957  9355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b4486f03fd9474aa0bb2cb4c9149437. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.881968  9356 tablet_service.cc:1467] Processing CreateTablet for tablet 1497b781f36c43b39d22a05bb524c59d (DEFAULT_TABLE table=rebalance_test_table_3 [id=a237d7bda970485480e98020c6ac738f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.882577  9356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1497b781f36c43b39d22a05bb524c59d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.890391 10414 tablet_bootstrap.cc:654] T 4674e376bf7440cda619784491685b44 P 525a093ea5bf40dc9bbf9dc39895d670: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.892556 10417 tablet_bootstrap.cc:492] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6: Bootstrap starting.
I20240502 21:28:54.895303 10417 tablet_bootstrap.cc:654] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.895888 10415 tablet_bootstrap.cc:492] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0: No bootstrap required, opened a new log
I20240502 21:28:54.896168 10415 ts_tablet_manager.cc:1367] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent bootstrapping tablet: real 0.019s	user 0.012s	sys 0.005s
I20240502 21:28:54.896406 10414 tablet_bootstrap.cc:492] T 4674e376bf7440cda619784491685b44 P 525a093ea5bf40dc9bbf9dc39895d670: No bootstrap required, opened a new log
I20240502 21:28:54.896598 10414 ts_tablet_manager.cc:1367] T 4674e376bf7440cda619784491685b44 P 525a093ea5bf40dc9bbf9dc39895d670: Time spent bootstrapping tablet: real 0.017s	user 0.004s	sys 0.000s
I20240502 21:28:54.897306 10415 raft_consensus.cc:348] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:54.897459 10414 raft_consensus.cc:348] T 4674e376bf7440cda619784491685b44 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:54.897598 10415 raft_consensus.cc:374] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.897686 10414 raft_consensus.cc:374] T 4674e376bf7440cda619784491685b44 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.897698 10415 raft_consensus.cc:729] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.897804 10414 raft_consensus.cc:729] T 4674e376bf7440cda619784491685b44 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.898020 10415 consensus_queue.cc:260] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:54.898082 10414 consensus_queue.cc:260] T 4674e376bf7440cda619784491685b44 P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:54.898697 10414 ts_tablet_manager.cc:1398] T 4674e376bf7440cda619784491685b44 P 525a093ea5bf40dc9bbf9dc39895d670: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 21:28:54.899031 10414 tablet_bootstrap.cc:492] T 1497b781f36c43b39d22a05bb524c59d P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap starting.
I20240502 21:28:54.900941 10414 tablet_bootstrap.cc:654] T 1497b781f36c43b39d22a05bb524c59d P 525a093ea5bf40dc9bbf9dc39895d670: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.901679 10419 tablet_bootstrap.cc:492] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58: Bootstrap starting.
I20240502 21:28:54.902000 10415 ts_tablet_manager.cc:1398] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.002s
I20240502 21:28:54.903455 10415 tablet_bootstrap.cc:492] T 8b4486f03fd9474aa0bb2cb4c9149437 P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap starting.
I20240502 21:28:54.903862 10419 tablet_bootstrap.cc:654] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.905483 10415 tablet_bootstrap.cc:654] T 8b4486f03fd9474aa0bb2cb4c9149437 P f2fdd596a8b44bbc848fe40f9f07fbe0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.907948 10417 tablet_bootstrap.cc:492] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6: No bootstrap required, opened a new log
I20240502 21:28:54.908140 10417 ts_tablet_manager.cc:1367] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6: Time spent bootstrapping tablet: real 0.016s	user 0.004s	sys 0.004s
I20240502 21:28:54.908571 10414 tablet_bootstrap.cc:492] T 1497b781f36c43b39d22a05bb524c59d P 525a093ea5bf40dc9bbf9dc39895d670: No bootstrap required, opened a new log
I20240502 21:28:54.908767 10414 ts_tablet_manager.cc:1367] T 1497b781f36c43b39d22a05bb524c59d P 525a093ea5bf40dc9bbf9dc39895d670: Time spent bootstrapping tablet: real 0.010s	user 0.003s	sys 0.000s
I20240502 21:28:54.908879 10415 tablet_bootstrap.cc:492] T 8b4486f03fd9474aa0bb2cb4c9149437 P f2fdd596a8b44bbc848fe40f9f07fbe0: No bootstrap required, opened a new log
I20240502 21:28:54.909046 10415 ts_tablet_manager.cc:1367] T 8b4486f03fd9474aa0bb2cb4c9149437 P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.002s
I20240502 21:28:54.909145 10417 raft_consensus.cc:348] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:54.909394 10417 raft_consensus.cc:374] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.909504 10417 raft_consensus.cc:729] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.909638 10415 raft_consensus.cc:348] T 8b4486f03fd9474aa0bb2cb4c9149437 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:54.909686 10414 raft_consensus.cc:348] T 1497b781f36c43b39d22a05bb524c59d P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:54.909859 10415 raft_consensus.cc:374] T 8b4486f03fd9474aa0bb2cb4c9149437 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.909914 10415 raft_consensus.cc:729] T 8b4486f03fd9474aa0bb2cb4c9149437 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.909772 10417 consensus_queue.cc:260] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:54.909917 10414 raft_consensus.cc:374] T 1497b781f36c43b39d22a05bb524c59d P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.910002 10414 raft_consensus.cc:729] T 1497b781f36c43b39d22a05bb524c59d P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.910130 10415 consensus_queue.cc:260] T 8b4486f03fd9474aa0bb2cb4c9149437 P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:54.910270 10414 consensus_queue.cc:260] T 1497b781f36c43b39d22a05bb524c59d P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:54.910696 10415 ts_tablet_manager.cc:1398] T 8b4486f03fd9474aa0bb2cb4c9149437 P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 21:28:54.911005 10415 tablet_bootstrap.cc:492] T 1497b781f36c43b39d22a05bb524c59d P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap starting.
I20240502 21:28:54.910957 10414 ts_tablet_manager.cc:1398] T 1497b781f36c43b39d22a05bb524c59d P 525a093ea5bf40dc9bbf9dc39895d670: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 21:28:54.911520 10419 tablet_bootstrap.cc:492] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58: No bootstrap required, opened a new log
I20240502 21:28:54.911844 10419 ts_tablet_manager.cc:1367] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58: Time spent bootstrapping tablet: real 0.010s	user 0.005s	sys 0.000s
I20240502 21:28:54.913111 10419 raft_consensus.cc:348] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:54.913544 10419 raft_consensus.cc:374] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.913601 10415 tablet_bootstrap.cc:654] T 1497b781f36c43b39d22a05bb524c59d P f2fdd596a8b44bbc848fe40f9f07fbe0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.913725 10419 raft_consensus.cc:729] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.914314 10419 consensus_queue.cc:260] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:54.915275 10417 ts_tablet_manager.cc:1398] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6: Time spent starting tablet: real 0.007s	user 0.001s	sys 0.006s
I20240502 21:28:54.915302 10419 ts_tablet_manager.cc:1398] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 21:28:54.915681 10419 tablet_bootstrap.cc:492] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58: Bootstrap starting.
I20240502 21:28:54.916443 10350 raft_consensus.cc:482] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:54.916534 10417 tablet_bootstrap.cc:492] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6: Bootstrap starting.
I20240502 21:28:54.916649 10350 raft_consensus.cc:504] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:54.917532 10350 leader_election.cc:288] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:28:54.918468 10417 tablet_bootstrap.cc:654] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.918493  9241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4674e376bf7440cda619784491685b44" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670" is_pre_election: true
I20240502 21:28:54.918825  9241 raft_consensus.cc:2454] T 4674e376bf7440cda619784491685b44 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 0.
I20240502 21:28:54.919072  9107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4674e376bf7440cda619784491685b44" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5543ddab5b484642b139b7c848932a58" is_pre_election: true
I20240502 21:28:54.919564  9313 leader_election.cc:302] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525a093ea5bf40dc9bbf9dc39895d670, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:28:54.919925 10350 raft_consensus.cc:2781] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:54.920060 10350 raft_consensus.cc:482] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:54.920146 10350 raft_consensus.cc:3037] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:28:54.924190  9310 leader_election.cc:341] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:54.924685 10419 tablet_bootstrap.cc:654] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.925066 10417 tablet_bootstrap.cc:492] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6: No bootstrap required, opened a new log
I20240502 21:28:54.925320 10417 ts_tablet_manager.cc:1367] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6: Time spent bootstrapping tablet: real 0.009s	user 0.007s	sys 0.000s
I20240502 21:28:54.925796 10350 raft_consensus.cc:504] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:54.926256 10417 raft_consensus.cc:348] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:54.926595 10417 raft_consensus.cc:374] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.926724 10417 raft_consensus.cc:729] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.927067 10417 consensus_queue.cc:260] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:54.928000  9241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4674e376bf7440cda619784491685b44" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670"
I20240502 21:28:54.928301  9241 raft_consensus.cc:3037] T 4674e376bf7440cda619784491685b44 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:54.934005  9241 raft_consensus.cc:2454] T 4674e376bf7440cda619784491685b44 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 1.
I20240502 21:28:54.934911  9313 leader_election.cc:302] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525a093ea5bf40dc9bbf9dc39895d670, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:28:54.939489  9107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4674e376bf7440cda619784491685b44" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5543ddab5b484642b139b7c848932a58"
W20240502 21:28:54.940611  9310 leader_election.cc:341] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:54.941326 10350 leader_election.cc:288] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 election: Requested vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:28:54.944252 10350 raft_consensus.cc:2781] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:54.944859 10350 raft_consensus.cc:686] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Becoming Leader. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Running, Role: LEADER
I20240502 21:28:54.945655 10350 consensus_queue.cc:237] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:54.954185 10415 tablet_bootstrap.cc:492] T 1497b781f36c43b39d22a05bb524c59d P f2fdd596a8b44bbc848fe40f9f07fbe0: No bootstrap required, opened a new log
I20240502 21:28:54.954432 10415 ts_tablet_manager.cc:1367] T 1497b781f36c43b39d22a05bb524c59d P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent bootstrapping tablet: real 0.044s	user 0.004s	sys 0.000s
I20240502 21:28:54.955868 10415 raft_consensus.cc:348] T 1497b781f36c43b39d22a05bb524c59d P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:54.956182 10415 raft_consensus.cc:374] T 1497b781f36c43b39d22a05bb524c59d P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.956328 10415 raft_consensus.cc:729] T 1497b781f36c43b39d22a05bb524c59d P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.956633 10415 consensus_queue.cc:260] T 1497b781f36c43b39d22a05bb524c59d P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:54.974658 10417 ts_tablet_manager.cc:1398] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6: Time spent starting tablet: real 0.049s	user 0.013s	sys 0.025s
I20240502 21:28:54.975242  8959 catalog_manager.cc:5484] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67). New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } }
I20240502 21:28:54.975986 10419 tablet_bootstrap.cc:492] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58: No bootstrap required, opened a new log
I20240502 21:28:54.976182 10419 ts_tablet_manager.cc:1367] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58: Time spent bootstrapping tablet: real 0.061s	user 0.034s	sys 0.017s
I20240502 21:28:54.977011 10419 raft_consensus.cc:348] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:54.977267 10419 raft_consensus.cc:374] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.977349 10419 raft_consensus.cc:729] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.977650 10419 consensus_queue.cc:260] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:54.978317 10419 ts_tablet_manager.cc:1398] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 21:28:54.979483 10415 ts_tablet_manager.cc:1398] T 1497b781f36c43b39d22a05bb524c59d P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent starting tablet: real 0.025s	user 0.005s	sys 0.006s
I20240502 21:28:55.028972 10428 raft_consensus.cc:482] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:55.029232 10428 raft_consensus.cc:504] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:55.030884 10428 leader_election.cc:288] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:28:55.048305  9376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1497b781f36c43b39d22a05bb524c59d" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" is_pre_election: true
I20240502 21:28:55.048565  9376 raft_consensus.cc:2454] T 1497b781f36c43b39d22a05bb524c59d P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19f162e164854f2da623a4bb8f9e6bf6 in term 0.
I20240502 21:28:55.049234  9241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1497b781f36c43b39d22a05bb524c59d" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670" is_pre_election: true
I20240502 21:28:55.049376  9446 leader_election.cc:302] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:28:55.049527  9241 raft_consensus.cc:2454] T 1497b781f36c43b39d22a05bb524c59d P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19f162e164854f2da623a4bb8f9e6bf6 in term 0.
I20240502 21:28:55.049899 10428 raft_consensus.cc:2781] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:55.050050 10428 raft_consensus.cc:482] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:55.050176 10428 raft_consensus.cc:3037] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:55.052477 10428 raft_consensus.cc:504] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:55.053162 10428 leader_election.cc:288] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 1 election: Requested vote from peers f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:28:55.053949  9241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1497b781f36c43b39d22a05bb524c59d" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670"
I20240502 21:28:55.054073  9376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1497b781f36c43b39d22a05bb524c59d" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
I20240502 21:28:55.054432  9376 raft_consensus.cc:3037] T 1497b781f36c43b39d22a05bb524c59d P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:55.054421  9241 raft_consensus.cc:3037] T 1497b781f36c43b39d22a05bb524c59d P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:55.057440  9376 raft_consensus.cc:2454] T 1497b781f36c43b39d22a05bb524c59d P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f162e164854f2da623a4bb8f9e6bf6 in term 1.
I20240502 21:28:55.057437  9241 raft_consensus.cc:2454] T 1497b781f36c43b39d22a05bb524c59d P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f162e164854f2da623a4bb8f9e6bf6 in term 1.
I20240502 21:28:55.058202  9446 leader_election.cc:302] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:28:55.058683 10428 raft_consensus.cc:2781] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:55.058864 10428 raft_consensus.cc:686] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 LEADER]: Becoming Leader. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Running, Role: LEADER
I20240502 21:28:55.059410 10428 consensus_queue.cc:237] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:55.062983  8958 catalog_manager.cc:5484] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68). New cstate: current_term: 1 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } }
I20240502 21:28:55.064306 10404 raft_consensus.cc:482] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:55.064491 10404 raft_consensus.cc:504] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:55.065245 10404 leader_election.cc:288] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
I20240502 21:28:55.065876  9376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b4486f03fd9474aa0bb2cb4c9149437" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" is_pre_election: true
I20240502 21:28:55.065912  9511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b4486f03fd9474aa0bb2cb4c9149437" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6" is_pre_election: true
I20240502 21:28:55.066155  9511 raft_consensus.cc:2454] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 0.
I20240502 21:28:55.066208  9376 raft_consensus.cc:2454] T 8b4486f03fd9474aa0bb2cb4c9149437 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 0.
I20240502 21:28:55.066658  9038 leader_election.cc:302] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 5543ddab5b484642b139b7c848932a58; no voters: 
I20240502 21:28:55.066984 10404 raft_consensus.cc:2781] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:55.067096 10404 raft_consensus.cc:482] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:55.067215 10404 raft_consensus.cc:3037] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:55.069192 10404 raft_consensus.cc:504] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:55.069875 10404 leader_election.cc:288] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 election: Requested vote from peers f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
I20240502 21:28:55.070466  9511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b4486f03fd9474aa0bb2cb4c9149437" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6"
I20240502 21:28:55.070674  9511 raft_consensus.cc:3037] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:55.070747  9376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b4486f03fd9474aa0bb2cb4c9149437" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
I20240502 21:28:55.070991  9376 raft_consensus.cc:3037] T 8b4486f03fd9474aa0bb2cb4c9149437 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:55.072810  9511 raft_consensus.cc:2454] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 1.
I20240502 21:28:55.072961  9376 raft_consensus.cc:2454] T 8b4486f03fd9474aa0bb2cb4c9149437 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 1.
I20240502 21:28:55.073347  9038 leader_election.cc:302] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 5543ddab5b484642b139b7c848932a58; no voters: 
I20240502 21:28:55.073681 10404 raft_consensus.cc:2781] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:55.074241 10404 raft_consensus.cc:686] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Becoming Leader. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Running, Role: LEADER
I20240502 21:28:55.074543 10404 consensus_queue.cc:237] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:55.078166  8958 catalog_manager.cc:5484] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5543ddab5b484642b139b7c848932a58 (127.27.196.65). New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } }
I20240502 21:28:55.119979 10404 raft_consensus.cc:482] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:55.120379 10404 raft_consensus.cc:504] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:55.121331 10404 leader_election.cc:288] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:28:55.122326  9376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4674e376bf7440cda619784491685b44" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" is_pre_election: true
I20240502 21:28:55.122330  9241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4674e376bf7440cda619784491685b44" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670" is_pre_election: true
I20240502 21:28:55.122805  9241 raft_consensus.cc:2379] T 4674e376bf7440cda619784491685b44 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5543ddab5b484642b139b7c848932a58 in current term 1: Already voted for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in this term.
I20240502 21:28:55.123484  9041 leader_election.cc:302] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5543ddab5b484642b139b7c848932a58; no voters: 525a093ea5bf40dc9bbf9dc39895d670, f2fdd596a8b44bbc848fe40f9f07fbe0
I20240502 21:28:55.123983 10404 raft_consensus.cc:3037] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:55.127256 10404 raft_consensus.cc:2726] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:28:55.332933 10432 consensus_queue.cc:1035] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:55.342531 10432 consensus_queue.cc:1035] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:55.453485 10428 consensus_queue.cc:1035] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:55.459331 10428 consensus_queue.cc:1035] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:55.486110 10404 consensus_queue.cc:1035] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:55.492846 10450 consensus_queue.cc:1035] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:28:55.524518 10440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:55.524951 10440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:55.533241 10440 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'

INSERT report
    rows total: 2000
    time total: 327.375 ms
  time per row: 0.163687 ms
I20240502 21:28:56.045188  8958 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54676:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 21:28:56.057029  9219 tablet_service.cc:1467] Processing CreateTablet for tablet dc1970b9263d41abb575b9d8639c59ae (DEFAULT_TABLE table=rebalance_test_table_4 [id=1b17408dbab9492bba0ed50efd6ef43f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.057253  9490 tablet_service.cc:1467] Processing CreateTablet for tablet dc1970b9263d41abb575b9d8639c59ae (DEFAULT_TABLE table=rebalance_test_table_4 [id=1b17408dbab9492bba0ed50efd6ef43f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.057695  9490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc1970b9263d41abb575b9d8639c59ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.057655  9219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc1970b9263d41abb575b9d8639c59ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.057657  9220 tablet_service.cc:1467] Processing CreateTablet for tablet cffcb4f7f7754262beeb06168097f6dc (DEFAULT_TABLE table=rebalance_test_table_4 [id=1b17408dbab9492bba0ed50efd6ef43f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.058449  9220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cffcb4f7f7754262beeb06168097f6dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.063901  9086 tablet_service.cc:1467] Processing CreateTablet for tablet dc1970b9263d41abb575b9d8639c59ae (DEFAULT_TABLE table=rebalance_test_table_4 [id=1b17408dbab9492bba0ed50efd6ef43f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.064597  9086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc1970b9263d41abb575b9d8639c59ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.067782 10511 tablet_bootstrap.cc:492] T cffcb4f7f7754262beeb06168097f6dc P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap starting.
I20240502 21:28:56.070866 10511 tablet_bootstrap.cc:654] T cffcb4f7f7754262beeb06168097f6dc P 525a093ea5bf40dc9bbf9dc39895d670: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.071377  9356 tablet_service.cc:1467] Processing CreateTablet for tablet cffcb4f7f7754262beeb06168097f6dc (DEFAULT_TABLE table=rebalance_test_table_4 [id=1b17408dbab9492bba0ed50efd6ef43f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.071828  9356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cffcb4f7f7754262beeb06168097f6dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.072376 10510 tablet_bootstrap.cc:492] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6: Bootstrap starting.
I20240502 21:28:56.073679 10511 tablet_bootstrap.cc:492] T cffcb4f7f7754262beeb06168097f6dc P 525a093ea5bf40dc9bbf9dc39895d670: No bootstrap required, opened a new log
I20240502 21:28:56.073907 10511 ts_tablet_manager.cc:1367] T cffcb4f7f7754262beeb06168097f6dc P 525a093ea5bf40dc9bbf9dc39895d670: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.003s
I20240502 21:28:56.074872 10510 tablet_bootstrap.cc:654] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.074872 10511 raft_consensus.cc:348] T cffcb4f7f7754262beeb06168097f6dc P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:56.075155 10511 raft_consensus.cc:374] T cffcb4f7f7754262beeb06168097f6dc P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.075273 10511 raft_consensus.cc:729] T cffcb4f7f7754262beeb06168097f6dc P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.075541 10511 consensus_queue.cc:260] T cffcb4f7f7754262beeb06168097f6dc P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:56.076296 10511 ts_tablet_manager.cc:1398] T cffcb4f7f7754262beeb06168097f6dc P 525a093ea5bf40dc9bbf9dc39895d670: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:28:56.076735 10511 tablet_bootstrap.cc:492] T dc1970b9263d41abb575b9d8639c59ae P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap starting.
I20240502 21:28:56.078121  9087 tablet_service.cc:1467] Processing CreateTablet for tablet cffcb4f7f7754262beeb06168097f6dc (DEFAULT_TABLE table=rebalance_test_table_4 [id=1b17408dbab9492bba0ed50efd6ef43f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.078761  9087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cffcb4f7f7754262beeb06168097f6dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.079350  9490 tablet_service.cc:1467] Processing CreateTablet for tablet f0259f01acb249dd99c662151dc21916 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1b17408dbab9492bba0ed50efd6ef43f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.079486 10511 tablet_bootstrap.cc:654] T dc1970b9263d41abb575b9d8639c59ae P 525a093ea5bf40dc9bbf9dc39895d670: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.079833  9490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0259f01acb249dd99c662151dc21916. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.082511  9085 tablet_service.cc:1467] Processing CreateTablet for tablet f0259f01acb249dd99c662151dc21916 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1b17408dbab9492bba0ed50efd6ef43f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.082954  9085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0259f01acb249dd99c662151dc21916. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.084178 10516 tablet_bootstrap.cc:492] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap starting.
I20240502 21:28:56.085551  9357 tablet_service.cc:1467] Processing CreateTablet for tablet f0259f01acb249dd99c662151dc21916 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1b17408dbab9492bba0ed50efd6ef43f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.085978  9357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0259f01acb249dd99c662151dc21916. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.086475 10516 tablet_bootstrap.cc:654] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.097827 10511 tablet_bootstrap.cc:492] T dc1970b9263d41abb575b9d8639c59ae P 525a093ea5bf40dc9bbf9dc39895d670: No bootstrap required, opened a new log
I20240502 21:28:56.098533 10511 ts_tablet_manager.cc:1367] T dc1970b9263d41abb575b9d8639c59ae P 525a093ea5bf40dc9bbf9dc39895d670: Time spent bootstrapping tablet: real 0.022s	user 0.005s	sys 0.004s
I20240502 21:28:56.099985 10511 raft_consensus.cc:348] T dc1970b9263d41abb575b9d8639c59ae P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:56.100203 10518 tablet_bootstrap.cc:492] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58: Bootstrap starting.
I20240502 21:28:56.100389 10511 raft_consensus.cc:374] T dc1970b9263d41abb575b9d8639c59ae P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.100582 10511 raft_consensus.cc:729] T dc1970b9263d41abb575b9d8639c59ae P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.100984 10511 consensus_queue.cc:260] T dc1970b9263d41abb575b9d8639c59ae P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:56.101702 10511 ts_tablet_manager.cc:1398] T dc1970b9263d41abb575b9d8639c59ae P 525a093ea5bf40dc9bbf9dc39895d670: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 21:28:56.102416 10518 tablet_bootstrap.cc:654] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.104315 10516 tablet_bootstrap.cc:492] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0: No bootstrap required, opened a new log
I20240502 21:28:56.104638 10516 ts_tablet_manager.cc:1367] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent bootstrapping tablet: real 0.020s	user 0.000s	sys 0.004s
I20240502 21:28:56.106392 10516 raft_consensus.cc:348] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:56.106643 10516 raft_consensus.cc:374] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.106804 10516 raft_consensus.cc:729] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.107220 10516 consensus_queue.cc:260] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:56.108110 10510 tablet_bootstrap.cc:492] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6: No bootstrap required, opened a new log
I20240502 21:28:56.108100 10516 ts_tablet_manager.cc:1398] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 21:28:56.108455 10516 tablet_bootstrap.cc:492] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap starting.
I20240502 21:28:56.108711 10510 ts_tablet_manager.cc:1367] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6: Time spent bootstrapping tablet: real 0.036s	user 0.013s	sys 0.009s
I20240502 21:28:56.109772 10510 raft_consensus.cc:348] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:56.110036 10510 raft_consensus.cc:374] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.110167 10510 raft_consensus.cc:729] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.110603 10516 tablet_bootstrap.cc:654] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.110564 10510 consensus_queue.cc:260] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:56.111552 10510 ts_tablet_manager.cc:1398] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 21:28:56.113013 10510 tablet_bootstrap.cc:492] T f0259f01acb249dd99c662151dc21916 P 19f162e164854f2da623a4bb8f9e6bf6: Bootstrap starting.
I20240502 21:28:56.115527 10516 tablet_bootstrap.cc:492] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0: No bootstrap required, opened a new log
I20240502 21:28:56.115780 10516 ts_tablet_manager.cc:1367] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent bootstrapping tablet: real 0.007s	user 0.004s	sys 0.000s
I20240502 21:28:56.116154 10518 tablet_bootstrap.cc:492] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58: No bootstrap required, opened a new log
I20240502 21:28:56.116323 10510 tablet_bootstrap.cc:654] T f0259f01acb249dd99c662151dc21916 P 19f162e164854f2da623a4bb8f9e6bf6: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.116510 10518 ts_tablet_manager.cc:1367] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58: Time spent bootstrapping tablet: real 0.016s	user 0.003s	sys 0.006s
I20240502 21:28:56.117090 10516 raft_consensus.cc:348] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:56.117452 10516 raft_consensus.cc:374] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.117604 10516 raft_consensus.cc:729] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.117524 10518 raft_consensus.cc:348] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:56.117817 10518 raft_consensus.cc:374] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.117969 10518 raft_consensus.cc:729] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.118002 10516 consensus_queue.cc:260] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:56.118335 10518 consensus_queue.cc:260] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:56.118672 10516 ts_tablet_manager.cc:1398] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 21:28:56.121145 10518 ts_tablet_manager.cc:1398] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240502 21:28:56.122893 10518 tablet_bootstrap.cc:492] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58: Bootstrap starting.
I20240502 21:28:56.125488 10518 tablet_bootstrap.cc:654] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.125777 10510 tablet_bootstrap.cc:492] T f0259f01acb249dd99c662151dc21916 P 19f162e164854f2da623a4bb8f9e6bf6: No bootstrap required, opened a new log
I20240502 21:28:56.126036 10510 ts_tablet_manager.cc:1367] T f0259f01acb249dd99c662151dc21916 P 19f162e164854f2da623a4bb8f9e6bf6: Time spent bootstrapping tablet: real 0.013s	user 0.009s	sys 0.000s
I20240502 21:28:56.126914 10510 raft_consensus.cc:348] T f0259f01acb249dd99c662151dc21916 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:56.127187 10510 raft_consensus.cc:374] T f0259f01acb249dd99c662151dc21916 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.127270 10510 raft_consensus.cc:729] T f0259f01acb249dd99c662151dc21916 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.127568 10510 consensus_queue.cc:260] T f0259f01acb249dd99c662151dc21916 P 19f162e164854f2da623a4bb8f9e6bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:56.127728 10518 tablet_bootstrap.cc:492] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58: No bootstrap required, opened a new log
I20240502 21:28:56.128010 10518 ts_tablet_manager.cc:1367] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240502 21:28:56.128206 10510 ts_tablet_manager.cc:1398] T f0259f01acb249dd99c662151dc21916 P 19f162e164854f2da623a4bb8f9e6bf6: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 21:28:56.128933 10518 raft_consensus.cc:348] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:56.129155 10518 raft_consensus.cc:374] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.129245 10518 raft_consensus.cc:729] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.129565 10518 consensus_queue.cc:260] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:56.130236 10518 ts_tablet_manager.cc:1398] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:28:56.130786 10518 tablet_bootstrap.cc:492] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58: Bootstrap starting.
I20240502 21:28:56.133003 10518 tablet_bootstrap.cc:654] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.135694 10518 tablet_bootstrap.cc:492] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58: No bootstrap required, opened a new log
I20240502 21:28:56.135893 10518 ts_tablet_manager.cc:1367] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.002s
I20240502 21:28:56.136590 10518 raft_consensus.cc:348] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:56.136780 10518 raft_consensus.cc:374] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.136888 10518 raft_consensus.cc:729] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.137118 10518 consensus_queue.cc:260] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:56.137650 10518 ts_tablet_manager.cc:1398] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 21:28:56.193200 10404 raft_consensus.cc:482] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:56.193419 10404 raft_consensus.cc:504] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:56.194181 10404 leader_election.cc:288] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:28:56.195093  9376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffcb4f7f7754262beeb06168097f6dc" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" is_pre_election: true
I20240502 21:28:56.195097  9242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffcb4f7f7754262beeb06168097f6dc" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670" is_pre_election: true
I20240502 21:28:56.195438  9376 raft_consensus.cc:2454] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 0.
I20240502 21:28:56.195438  9242 raft_consensus.cc:2454] T cffcb4f7f7754262beeb06168097f6dc P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 0.
I20240502 21:28:56.196003  9040 leader_election.cc:302] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5543ddab5b484642b139b7c848932a58, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:28:56.196360 10404 raft_consensus.cc:2781] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:56.196465 10404 raft_consensus.cc:482] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:56.196576 10404 raft_consensus.cc:3037] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.198648 10450 raft_consensus.cc:482] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:56.198872 10450 raft_consensus.cc:504] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:56.198931 10404 raft_consensus.cc:504] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:56.199786 10404 leader_election.cc:288] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 election: Requested vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:28:56.199734 10450 leader_election.cc:288] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:28:56.200495  9376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0259f01acb249dd99c662151dc21916" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" is_pre_election: true
I20240502 21:28:56.200603  9378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffcb4f7f7754262beeb06168097f6dc" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
I20240502 21:28:56.200663  9511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0259f01acb249dd99c662151dc21916" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6" is_pre_election: true
I20240502 21:28:56.200789  9378 raft_consensus.cc:3037] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.200831  9376 raft_consensus.cc:2454] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 0.
I20240502 21:28:56.201036  9511 raft_consensus.cc:2454] T f0259f01acb249dd99c662151dc21916 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 0.
I20240502 21:28:56.201539  9242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffcb4f7f7754262beeb06168097f6dc" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670"
I20240502 21:28:56.201606  9040 leader_election.cc:302] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 5543ddab5b484642b139b7c848932a58, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:28:56.201804  9242 raft_consensus.cc:3037] T cffcb4f7f7754262beeb06168097f6dc P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.201918 10404 raft_consensus.cc:482] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:56.202124 10450 raft_consensus.cc:2781] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:56.202236 10450 raft_consensus.cc:482] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:56.202152 10404 raft_consensus.cc:504] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:56.202342 10450 raft_consensus.cc:3037] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.203943  9378 raft_consensus.cc:2454] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 1.
I20240502 21:28:56.204113 10404 leader_election.cc:288] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
I20240502 21:28:56.204478  9040 leader_election.cc:302] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5543ddab5b484642b139b7c848932a58, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:28:56.204783  9242 raft_consensus.cc:2454] T cffcb4f7f7754262beeb06168097f6dc P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 1.
I20240502 21:28:56.204829 10404 raft_consensus.cc:2781] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:56.205076 10404 raft_consensus.cc:686] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Becoming Leader. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Running, Role: LEADER
I20240502 21:28:56.205473 10404 consensus_queue.cc:237] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:56.205626 10450 raft_consensus.cc:504] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:56.206310 10450 leader_election.cc:288] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 election: Requested vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:28:56.206763  9241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc1970b9263d41abb575b9d8639c59ae" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670" is_pre_election: true
I20240502 21:28:56.206894  9378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0259f01acb249dd99c662151dc21916" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
I20240502 21:28:56.206902  9508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0259f01acb249dd99c662151dc21916" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6"
I20240502 21:28:56.207211  9508 raft_consensus.cc:3037] T f0259f01acb249dd99c662151dc21916 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.207199  9378 raft_consensus.cc:3037] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.207160  9241 raft_consensus.cc:2454] T dc1970b9263d41abb575b9d8639c59ae P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 0.
I20240502 21:28:56.207756  9511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc1970b9263d41abb575b9d8639c59ae" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6" is_pre_election: true
I20240502 21:28:56.208137  9511 raft_consensus.cc:2454] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 0.
I20240502 21:28:56.210186  9378 raft_consensus.cc:2454] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 1.
I20240502 21:28:56.210283  9508 raft_consensus.cc:2454] T f0259f01acb249dd99c662151dc21916 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 1.
I20240502 21:28:56.210943  9038 leader_election.cc:302] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 525a093ea5bf40dc9bbf9dc39895d670, 5543ddab5b484642b139b7c848932a58; no voters: 
I20240502 21:28:56.211144  9040 leader_election.cc:302] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5543ddab5b484642b139b7c848932a58, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:28:56.211439 10404 raft_consensus.cc:2781] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:56.211575 10404 raft_consensus.cc:482] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:56.211661 10404 raft_consensus.cc:3037] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.211359  8956 catalog_manager.cc:5484] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5543ddab5b484642b139b7c848932a58 (127.27.196.65). New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } }
I20240502 21:28:56.211828 10450 raft_consensus.cc:2781] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:56.212026 10450 raft_consensus.cc:686] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Becoming Leader. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Running, Role: LEADER
I20240502 21:28:56.212332 10450 consensus_queue.cc:237] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:56.214347 10404 raft_consensus.cc:504] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:56.215168 10404 leader_election.cc:288] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 election: Requested vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
I20240502 21:28:56.216327  9508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc1970b9263d41abb575b9d8639c59ae" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6"
I20240502 21:28:56.216361  9241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc1970b9263d41abb575b9d8639c59ae" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670"
I20240502 21:28:56.216615  9508 raft_consensus.cc:3037] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.216638  9241 raft_consensus.cc:3037] T dc1970b9263d41abb575b9d8639c59ae P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.218797  9508 raft_consensus.cc:2454] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 1.
I20240502 21:28:56.219290  9038 leader_election.cc:302] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 5543ddab5b484642b139b7c848932a58; no voters: 
I20240502 21:28:56.219641 10404 raft_consensus.cc:2781] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:56.219818 10404 raft_consensus.cc:686] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Becoming Leader. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Running, Role: LEADER
I20240502 21:28:56.220162 10404 consensus_queue.cc:237] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:56.221117  9241 raft_consensus.cc:2454] T dc1970b9263d41abb575b9d8639c59ae P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 1.
I20240502 21:28:56.224289  8956 catalog_manager.cc:5484] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5543ddab5b484642b139b7c848932a58 (127.27.196.65). New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } }
I20240502 21:28:56.232103  8956 catalog_manager.cc:5484] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5543ddab5b484642b139b7c848932a58 (127.27.196.65). New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } }
I20240502 21:28:56.635282 10404 consensus_queue.cc:1035] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:56.653592 10450 consensus_queue.cc:1035] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:56.656669 10404 consensus_queue.cc:1035] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:56.668638 10404 consensus_queue.cc:1035] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:56.675413 10404 consensus_queue.cc:1035] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:56.688012 10438 consensus_queue.cc:1035] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:28:56.717121 10528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:56.717536 10528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:56.725940 10528 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'

INSERT report
    rows total: 2000
    time total: 274.214 ms
  time per row: 0.137107 ms
I20240502 21:28:57.182574  8963 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54676:
name: "rebalance_test_table_5"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 21:28:57.195149  9490 tablet_service.cc:1467] Processing CreateTablet for tablet 8a2e06253c51405fa9b6bc9e87844760 (DEFAULT_TABLE table=rebalance_test_table_5 [id=e8eef05f850243d0bdc7e1899f8c6139]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:57.195535  9220 tablet_service.cc:1467] Processing CreateTablet for tablet 8a2e06253c51405fa9b6bc9e87844760 (DEFAULT_TABLE table=rebalance_test_table_5 [id=e8eef05f850243d0bdc7e1899f8c6139]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:57.195535  9356 tablet_service.cc:1467] Processing CreateTablet for tablet 8a2e06253c51405fa9b6bc9e87844760 (DEFAULT_TABLE table=rebalance_test_table_5 [id=e8eef05f850243d0bdc7e1899f8c6139]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:57.195778  9490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2e06253c51405fa9b6bc9e87844760. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:57.196151  9220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2e06253c51405fa9b6bc9e87844760. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:57.196162  9356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2e06253c51405fa9b6bc9e87844760. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:57.196522  9219 tablet_service.cc:1467] Processing CreateTablet for tablet f82c35077224414cb284472d49d7de4f (DEFAULT_TABLE table=rebalance_test_table_5 [id=e8eef05f850243d0bdc7e1899f8c6139]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:57.197032  9219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f82c35077224414cb284472d49d7de4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:57.203482  9489 tablet_service.cc:1467] Processing CreateTablet for tablet f82c35077224414cb284472d49d7de4f (DEFAULT_TABLE table=rebalance_test_table_5 [id=e8eef05f850243d0bdc7e1899f8c6139]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:57.204013  9489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f82c35077224414cb284472d49d7de4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:57.206338  9221 tablet_service.cc:1467] Processing CreateTablet for tablet 32c306d724954a16be21a34aa4cc0b5f (DEFAULT_TABLE table=rebalance_test_table_5 [id=e8eef05f850243d0bdc7e1899f8c6139]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:57.206799  9221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32c306d724954a16be21a34aa4cc0b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:57.206524  9357 tablet_service.cc:1467] Processing CreateTablet for tablet f82c35077224414cb284472d49d7de4f (DEFAULT_TABLE table=rebalance_test_table_5 [id=e8eef05f850243d0bdc7e1899f8c6139]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:57.207504  9357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f82c35077224414cb284472d49d7de4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:57.214375  9086 tablet_service.cc:1467] Processing CreateTablet for tablet 32c306d724954a16be21a34aa4cc0b5f (DEFAULT_TABLE table=rebalance_test_table_5 [id=e8eef05f850243d0bdc7e1899f8c6139]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:57.214915  9086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32c306d724954a16be21a34aa4cc0b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:57.218415  9489 tablet_service.cc:1467] Processing CreateTablet for tablet 32c306d724954a16be21a34aa4cc0b5f (DEFAULT_TABLE table=rebalance_test_table_5 [id=e8eef05f850243d0bdc7e1899f8c6139]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:57.219079  9489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32c306d724954a16be21a34aa4cc0b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:57.219709 10590 tablet_bootstrap.cc:492] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap starting.
I20240502 21:28:57.222173 10590 tablet_bootstrap.cc:654] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:57.227375 10592 tablet_bootstrap.cc:492] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58: Bootstrap starting.
I20240502 21:28:57.230990 10592 tablet_bootstrap.cc:654] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:57.236521 10589 tablet_bootstrap.cc:492] T 8a2e06253c51405fa9b6bc9e87844760 P 19f162e164854f2da623a4bb8f9e6bf6: Bootstrap starting.
I20240502 21:28:57.241744 10589 tablet_bootstrap.cc:654] T 8a2e06253c51405fa9b6bc9e87844760 P 19f162e164854f2da623a4bb8f9e6bf6: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:57.241851 10591 tablet_bootstrap.cc:492] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap starting.
I20240502 21:28:57.244303 10590 tablet_bootstrap.cc:492] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0: No bootstrap required, opened a new log
I20240502 21:28:57.244598 10590 ts_tablet_manager.cc:1367] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent bootstrapping tablet: real 0.025s	user 0.014s	sys 0.000s
I20240502 21:28:57.245733 10592 tablet_bootstrap.cc:492] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58: No bootstrap required, opened a new log
I20240502 21:28:57.245916 10592 ts_tablet_manager.cc:1367] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58: Time spent bootstrapping tablet: real 0.019s	user 0.008s	sys 0.004s
I20240502 21:28:57.245805 10590 raft_consensus.cc:348] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:57.246028 10590 raft_consensus.cc:374] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:57.246201 10590 raft_consensus.cc:729] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Initialized, Role: FOLLOWER
I20240502 21:28:57.246731 10590 consensus_queue.cc:260] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:57.246963 10592 raft_consensus.cc:348] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:57.247272 10592 raft_consensus.cc:374] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:57.247399 10592 raft_consensus.cc:729] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Initialized, Role: FOLLOWER
I20240502 21:28:57.247689 10591 tablet_bootstrap.cc:654] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:57.247728 10592 consensus_queue.cc:260] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:57.248391 10592 ts_tablet_manager.cc:1398] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 21:28:57.249493 10589 tablet_bootstrap.cc:492] T 8a2e06253c51405fa9b6bc9e87844760 P 19f162e164854f2da623a4bb8f9e6bf6: No bootstrap required, opened a new log
I20240502 21:28:57.251144 10589 ts_tablet_manager.cc:1367] T 8a2e06253c51405fa9b6bc9e87844760 P 19f162e164854f2da623a4bb8f9e6bf6: Time spent bootstrapping tablet: real 0.015s	user 0.005s	sys 0.000s
I20240502 21:28:57.252478 10589 raft_consensus.cc:348] T 8a2e06253c51405fa9b6bc9e87844760 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:57.252868 10589 raft_consensus.cc:374] T 8a2e06253c51405fa9b6bc9e87844760 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:57.253047 10589 raft_consensus.cc:729] T 8a2e06253c51405fa9b6bc9e87844760 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Initialized, Role: FOLLOWER
I20240502 21:28:57.254737 10590 ts_tablet_manager.cc:1398] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.000s
I20240502 21:28:57.253480 10589 consensus_queue.cc:260] T 8a2e06253c51405fa9b6bc9e87844760 P 19f162e164854f2da623a4bb8f9e6bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:57.256101 10589 ts_tablet_manager.cc:1398] T 8a2e06253c51405fa9b6bc9e87844760 P 19f162e164854f2da623a4bb8f9e6bf6: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.002s
I20240502 21:28:57.256497 10589 tablet_bootstrap.cc:492] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6: Bootstrap starting.
I20240502 21:28:57.257817 10590 tablet_bootstrap.cc:492] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap starting.
I20240502 21:28:57.258337 10589 tablet_bootstrap.cc:654] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:57.261171 10591 tablet_bootstrap.cc:492] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670: No bootstrap required, opened a new log
I20240502 21:28:57.261866 10590 tablet_bootstrap.cc:654] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:57.261873 10591 ts_tablet_manager.cc:1367] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670: Time spent bootstrapping tablet: real 0.020s	user 0.011s	sys 0.000s
I20240502 21:28:57.263002 10591 raft_consensus.cc:348] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:57.263484 10591 raft_consensus.cc:374] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:57.263680 10591 raft_consensus.cc:729] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Initialized, Role: FOLLOWER
I20240502 21:28:57.264124 10591 consensus_queue.cc:260] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:57.266185 10591 ts_tablet_manager.cc:1398] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 21:28:57.266714 10591 tablet_bootstrap.cc:492] T 32c306d724954a16be21a34aa4cc0b5f P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap starting.
I20240502 21:28:57.270242 10591 tablet_bootstrap.cc:654] T 32c306d724954a16be21a34aa4cc0b5f P 525a093ea5bf40dc9bbf9dc39895d670: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:57.274681 10590 tablet_bootstrap.cc:492] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0: No bootstrap required, opened a new log
I20240502 21:28:57.274838 10589 tablet_bootstrap.cc:492] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6: No bootstrap required, opened a new log
I20240502 21:28:57.274924 10590 ts_tablet_manager.cc:1367] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent bootstrapping tablet: real 0.017s	user 0.002s	sys 0.008s
I20240502 21:28:57.275022 10591 tablet_bootstrap.cc:492] T 32c306d724954a16be21a34aa4cc0b5f P 525a093ea5bf40dc9bbf9dc39895d670: No bootstrap required, opened a new log
I20240502 21:28:57.275039 10589 ts_tablet_manager.cc:1367] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6: Time spent bootstrapping tablet: real 0.019s	user 0.004s	sys 0.011s
I20240502 21:28:57.275249 10591 ts_tablet_manager.cc:1367] T 32c306d724954a16be21a34aa4cc0b5f P 525a093ea5bf40dc9bbf9dc39895d670: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.004s
I20240502 21:28:57.275751 10589 raft_consensus.cc:348] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:57.275949 10589 raft_consensus.cc:374] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:57.275861 10590 raft_consensus.cc:348] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:57.276032 10589 raft_consensus.cc:729] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Initialized, Role: FOLLOWER
I20240502 21:28:57.276103 10590 raft_consensus.cc:374] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:57.276050 10591 raft_consensus.cc:348] T 32c306d724954a16be21a34aa4cc0b5f P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:57.276189 10590 raft_consensus.cc:729] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Initialized, Role: FOLLOWER
I20240502 21:28:57.276295 10591 raft_consensus.cc:374] T 32c306d724954a16be21a34aa4cc0b5f P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:57.276322 10589 consensus_queue.cc:260] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:57.276458 10591 raft_consensus.cc:729] T 32c306d724954a16be21a34aa4cc0b5f P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Initialized, Role: FOLLOWER
I20240502 21:28:57.276464 10590 consensus_queue.cc:260] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:57.276926 10589 ts_tablet_manager.cc:1398] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 21:28:57.277050 10590 ts_tablet_manager.cc:1398] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 21:28:57.277365 10589 tablet_bootstrap.cc:492] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6: Bootstrap starting.
I20240502 21:28:57.277382 10591 consensus_queue.cc:260] T 32c306d724954a16be21a34aa4cc0b5f P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:57.277976 10591 ts_tablet_manager.cc:1398] T 32c306d724954a16be21a34aa4cc0b5f P 525a093ea5bf40dc9bbf9dc39895d670: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240502 21:28:57.278281 10591 tablet_bootstrap.cc:492] T f82c35077224414cb284472d49d7de4f P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap starting.
I20240502 21:28:57.279430 10589 tablet_bootstrap.cc:654] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:57.280052 10591 tablet_bootstrap.cc:654] T f82c35077224414cb284472d49d7de4f P 525a093ea5bf40dc9bbf9dc39895d670: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:57.284441 10591 tablet_bootstrap.cc:492] T f82c35077224414cb284472d49d7de4f P 525a093ea5bf40dc9bbf9dc39895d670: No bootstrap required, opened a new log
I20240502 21:28:57.284615 10591 ts_tablet_manager.cc:1367] T f82c35077224414cb284472d49d7de4f P 525a093ea5bf40dc9bbf9dc39895d670: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
I20240502 21:28:57.284759 10589 tablet_bootstrap.cc:492] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6: No bootstrap required, opened a new log
I20240502 21:28:57.284914 10589 ts_tablet_manager.cc:1367] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.001s
I20240502 21:28:57.285493 10591 raft_consensus.cc:348] T f82c35077224414cb284472d49d7de4f P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:57.285727 10591 raft_consensus.cc:374] T f82c35077224414cb284472d49d7de4f P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:57.285620 10589 raft_consensus.cc:348] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:57.285816 10589 raft_consensus.cc:374] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:57.285828 10591 raft_consensus.cc:729] T f82c35077224414cb284472d49d7de4f P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Initialized, Role: FOLLOWER
I20240502 21:28:57.285917 10589 raft_consensus.cc:729] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Initialized, Role: FOLLOWER
I20240502 21:28:57.286106 10591 consensus_queue.cc:260] T f82c35077224414cb284472d49d7de4f P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:57.286141 10589 consensus_queue.cc:260] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:57.286661 10589 ts_tablet_manager.cc:1398] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 21:28:57.286664 10591 ts_tablet_manager.cc:1398] T f82c35077224414cb284472d49d7de4f P 525a093ea5bf40dc9bbf9dc39895d670: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:28:57.319356 10438 raft_consensus.cc:482] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:57.319698 10438 raft_consensus.cc:504] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:57.320536 10438 leader_election.cc:288] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
I20240502 21:28:57.321074  9242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c306d724954a16be21a34aa4cc0b5f" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670" is_pre_election: true
I20240502 21:28:57.321365  9242 raft_consensus.cc:2454] T 32c306d724954a16be21a34aa4cc0b5f P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 0.
I20240502 21:28:57.321374  9511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c306d724954a16be21a34aa4cc0b5f" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6" is_pre_election: true
I20240502 21:28:57.321725  9511 raft_consensus.cc:2454] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 0.
I20240502 21:28:57.321964  9041 leader_election.cc:302] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525a093ea5bf40dc9bbf9dc39895d670, 5543ddab5b484642b139b7c848932a58; no voters: 
I20240502 21:28:57.322350 10438 raft_consensus.cc:2781] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:57.322450 10438 raft_consensus.cc:482] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:57.322582 10438 raft_consensus.cc:3037] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:57.325089 10438 raft_consensus.cc:504] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:57.325832 10438 leader_election.cc:288] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 election: Requested vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
I20240502 21:28:57.326433  9511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c306d724954a16be21a34aa4cc0b5f" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6"
I20240502 21:28:57.326529  9242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c306d724954a16be21a34aa4cc0b5f" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670"
I20240502 21:28:57.326701  9511 raft_consensus.cc:3037] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:57.326779  9242 raft_consensus.cc:3037] T 32c306d724954a16be21a34aa4cc0b5f P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:57.329337  9511 raft_consensus.cc:2454] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 1.
I20240502 21:28:57.329370  9242 raft_consensus.cc:2454] T 32c306d724954a16be21a34aa4cc0b5f P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 1.
I20240502 21:28:57.329913  9038 leader_election.cc:302] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 5543ddab5b484642b139b7c848932a58; no voters: 
I20240502 21:28:57.330273 10438 raft_consensus.cc:2781] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:57.330572 10438 raft_consensus.cc:686] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Becoming Leader. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Running, Role: LEADER
I20240502 21:28:57.330925 10438 consensus_queue.cc:237] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:57.334545  8963 catalog_manager.cc:5484] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5543ddab5b484642b139b7c848932a58 (127.27.196.65). New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } }
I20240502 21:28:57.360906 10431 raft_consensus.cc:482] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:57.361171 10431 raft_consensus.cc:504] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:57.362054 10431 leader_election.cc:288] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:28:57.362794  9242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2e06253c51405fa9b6bc9e87844760" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670" is_pre_election: true
I20240502 21:28:57.362823  9511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2e06253c51405fa9b6bc9e87844760" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6" is_pre_election: true
I20240502 21:28:57.363170  9242 raft_consensus.cc:2454] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 0.
I20240502 21:28:57.363206  9511 raft_consensus.cc:2454] T 8a2e06253c51405fa9b6bc9e87844760 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 0.
I20240502 21:28:57.363754  9313 leader_election.cc:302] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525a093ea5bf40dc9bbf9dc39895d670, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:28:57.364225 10431 raft_consensus.cc:2781] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:57.364416 10431 raft_consensus.cc:482] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:57.364533 10431 raft_consensus.cc:3037] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:57.367345 10431 raft_consensus.cc:504] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:57.368127 10431 leader_election.cc:288] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 election: Requested vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:28:57.368896  9242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2e06253c51405fa9b6bc9e87844760" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670"
I20240502 21:28:57.368925  9511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2e06253c51405fa9b6bc9e87844760" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6"
I20240502 21:28:57.369194  9511 raft_consensus.cc:3037] T 8a2e06253c51405fa9b6bc9e87844760 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:57.369239  9242 raft_consensus.cc:3037] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:57.372388  9511 raft_consensus.cc:2454] T 8a2e06253c51405fa9b6bc9e87844760 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 1.
I20240502 21:28:57.372373  9242 raft_consensus.cc:2454] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 1.
I20240502 21:28:57.373028  9310 leader_election.cc:302] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 525a093ea5bf40dc9bbf9dc39895d670, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:28:57.373453 10431 raft_consensus.cc:2781] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:57.373958 10431 raft_consensus.cc:686] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Becoming Leader. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Running, Role: LEADER
I20240502 21:28:57.374467 10431 consensus_queue.cc:237] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:57.378368  8956 catalog_manager.cc:5484] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67). New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } }
I20240502 21:28:57.551388 10424 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:57.551648 10424 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:57.552433 10424 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:28:57.553396  9378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f82c35077224414cb284472d49d7de4f" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" is_pre_election: true
I20240502 21:28:57.553409  9242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f82c35077224414cb284472d49d7de4f" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670" is_pre_election: true
I20240502 21:28:57.553697  9378 raft_consensus.cc:2454] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19f162e164854f2da623a4bb8f9e6bf6 in term 0.
I20240502 21:28:57.553697  9242 raft_consensus.cc:2454] T f82c35077224414cb284472d49d7de4f P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19f162e164854f2da623a4bb8f9e6bf6 in term 0.
I20240502 21:28:57.554301  9446 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:28:57.554726 10424 raft_consensus.cc:2781] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:57.554965 10424 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:57.555155 10424 raft_consensus.cc:3037] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:57.557776 10424 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:57.558627 10424 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 1 election: Requested vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:28:57.559469  9242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f82c35077224414cb284472d49d7de4f" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670"
I20240502 21:28:57.559587  9378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f82c35077224414cb284472d49d7de4f" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
I20240502 21:28:57.559795  9378 raft_consensus.cc:3037] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:57.559787  9242 raft_consensus.cc:3037] T f82c35077224414cb284472d49d7de4f P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:57.562975  9242 raft_consensus.cc:2454] T f82c35077224414cb284472d49d7de4f P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f162e164854f2da623a4bb8f9e6bf6 in term 1.
I20240502 21:28:57.562949  9378 raft_consensus.cc:2454] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f162e164854f2da623a4bb8f9e6bf6 in term 1.
I20240502 21:28:57.563724  9447 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 525a093ea5bf40dc9bbf9dc39895d670; no voters: 
I20240502 21:28:57.564203 10424 raft_consensus.cc:2781] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:57.564656 10424 raft_consensus.cc:686] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 LEADER]: Becoming Leader. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Running, Role: LEADER
I20240502 21:28:57.565142 10424 consensus_queue.cc:237] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:57.569396  8956 catalog_manager.cc:5484] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68). New cstate: current_term: 1 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } }
I20240502 21:28:57.771754 10438 consensus_queue.cc:1035] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:57.780002 10404 consensus_queue.cc:1035] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:57.906358 10604 consensus_queue.cc:1035] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:57.955942 10616 consensus_queue.cc:1035] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:28:58.028254 10609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:58.028679 10609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:58.037580 10609 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:28:58.043536 10424 consensus_queue.cc:1035] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:58.048918 10607 consensus_queue.cc:1035] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'

INSERT report
    rows total: 2000
    time total: 279.765 ms
  time per row: 0.139882 ms
I20240502 21:28:58.511504  8954 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54676:
name: "rebalance_test_table_6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 21:28:58.522085  9357 tablet_service.cc:1467] Processing CreateTablet for tablet a7c6b881c9ef4cdca787f065778e526f (DEFAULT_TABLE table=rebalance_test_table_6 [id=265d5656d7c048aa9795a906661b8ea8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:58.522631  9357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7c6b881c9ef4cdca787f065778e526f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:58.522889  9086 tablet_service.cc:1467] Processing CreateTablet for tablet a7c6b881c9ef4cdca787f065778e526f (DEFAULT_TABLE table=rebalance_test_table_6 [id=265d5656d7c048aa9795a906661b8ea8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:58.522830  9220 tablet_service.cc:1467] Processing CreateTablet for tablet a7c6b881c9ef4cdca787f065778e526f (DEFAULT_TABLE table=rebalance_test_table_6 [id=265d5656d7c048aa9795a906661b8ea8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:58.523497  9220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7c6b881c9ef4cdca787f065778e526f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:58.524483  9490 tablet_service.cc:1467] Processing CreateTablet for tablet be34ec89876d42b69efe1878c0d9bc19 (DEFAULT_TABLE table=rebalance_test_table_6 [id=265d5656d7c048aa9795a906661b8ea8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:58.524758  9087 tablet_service.cc:1467] Processing CreateTablet for tablet be34ec89876d42b69efe1878c0d9bc19 (DEFAULT_TABLE table=rebalance_test_table_6 [id=265d5656d7c048aa9795a906661b8ea8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:58.525069  9490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be34ec89876d42b69efe1878c0d9bc19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:58.525389  9085 tablet_service.cc:1467] Processing CreateTablet for tablet f9ae8fac693d4d68b6741f2509e1265e (DEFAULT_TABLE table=rebalance_test_table_6 [id=265d5656d7c048aa9795a906661b8ea8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:58.525976  9086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7c6b881c9ef4cdca787f065778e526f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:58.527810  9087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be34ec89876d42b69efe1878c0d9bc19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:58.530148  9219 tablet_service.cc:1467] Processing CreateTablet for tablet f9ae8fac693d4d68b6741f2509e1265e (DEFAULT_TABLE table=rebalance_test_table_6 [id=265d5656d7c048aa9795a906661b8ea8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:58.530584  9219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9ae8fac693d4d68b6741f2509e1265e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:58.530580  9356 tablet_service.cc:1467] Processing CreateTablet for tablet be34ec89876d42b69efe1878c0d9bc19 (DEFAULT_TABLE table=rebalance_test_table_6 [id=265d5656d7c048aa9795a906661b8ea8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:58.531162  9356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be34ec89876d42b69efe1878c0d9bc19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:58.536054  9355 tablet_service.cc:1467] Processing CreateTablet for tablet f9ae8fac693d4d68b6741f2509e1265e (DEFAULT_TABLE table=rebalance_test_table_6 [id=265d5656d7c048aa9795a906661b8ea8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:58.536793  9355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9ae8fac693d4d68b6741f2509e1265e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:58.537176  9085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9ae8fac693d4d68b6741f2509e1265e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:58.554383 10680 tablet_bootstrap.cc:492] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap starting.
I20240502 21:28:58.558375 10679 tablet_bootstrap.cc:492] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6: Bootstrap starting.
I20240502 21:28:58.559319 10680 tablet_bootstrap.cc:654] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:58.560645 10679 tablet_bootstrap.cc:654] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:58.571583 10679 tablet_bootstrap.cc:492] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6: No bootstrap required, opened a new log
I20240502 21:28:58.572067 10679 ts_tablet_manager.cc:1367] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6: Time spent bootstrapping tablet: real 0.014s	user 0.006s	sys 0.005s
I20240502 21:28:58.573518 10679 raft_consensus.cc:348] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:58.574112 10679 raft_consensus.cc:374] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:58.574307 10679 raft_consensus.cc:729] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Initialized, Role: FOLLOWER
I20240502 21:28:58.575805 10679 consensus_queue.cc:260] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:58.577711 10679 ts_tablet_manager.cc:1398] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 21:28:58.582423 10678 tablet_bootstrap.cc:492] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap starting.
I20240502 21:28:58.585309 10678 tablet_bootstrap.cc:654] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:58.586531 10683 tablet_bootstrap.cc:492] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58: Bootstrap starting.
I20240502 21:28:58.588008 10680 tablet_bootstrap.cc:492] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670: No bootstrap required, opened a new log
I20240502 21:28:58.588213 10680 ts_tablet_manager.cc:1367] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670: Time spent bootstrapping tablet: real 0.034s	user 0.011s	sys 0.016s
I20240502 21:28:58.588757 10683 tablet_bootstrap.cc:654] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:58.588765 10678 tablet_bootstrap.cc:492] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0: No bootstrap required, opened a new log
I20240502 21:28:58.588948 10678 ts_tablet_manager.cc:1367] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.000s
I20240502 21:28:58.589335 10680 raft_consensus.cc:348] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:58.589603 10680 raft_consensus.cc:374] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:58.589649 10678 raft_consensus.cc:348] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:58.589764 10680 raft_consensus.cc:729] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Initialized, Role: FOLLOWER
I20240502 21:28:58.589866 10678 raft_consensus.cc:374] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:58.590040 10678 raft_consensus.cc:729] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Initialized, Role: FOLLOWER
I20240502 21:28:58.590191 10680 consensus_queue.cc:260] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:58.590386 10678 consensus_queue.cc:260] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:58.590920 10680 ts_tablet_manager.cc:1398] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240502 21:28:58.591097 10678 ts_tablet_manager.cc:1398] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:28:58.591388 10680 tablet_bootstrap.cc:492] T f9ae8fac693d4d68b6741f2509e1265e P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap starting.
I20240502 21:28:58.591583 10683 tablet_bootstrap.cc:492] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58: No bootstrap required, opened a new log
I20240502 21:28:58.591820 10683 ts_tablet_manager.cc:1367] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58: Time spent bootstrapping tablet: real 0.005s	user 0.005s	sys 0.000s
I20240502 21:28:58.592773 10683 raft_consensus.cc:348] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:58.593025 10683 raft_consensus.cc:374] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:58.593143 10683 raft_consensus.cc:729] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Initialized, Role: FOLLOWER
I20240502 21:28:58.593539 10683 consensus_queue.cc:260] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:58.594157 10678 tablet_bootstrap.cc:492] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap starting.
I20240502 21:28:58.596941 10683 ts_tablet_manager.cc:1398] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 21:28:58.598207 10678 tablet_bootstrap.cc:654] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:58.598721 10683 tablet_bootstrap.cc:492] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58: Bootstrap starting.
I20240502 21:28:58.600615 10683 tablet_bootstrap.cc:654] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:58.600664 10678 tablet_bootstrap.cc:492] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0: No bootstrap required, opened a new log
I20240502 21:28:58.600831 10678 ts_tablet_manager.cc:1367] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.000s
I20240502 21:28:58.601114 10680 tablet_bootstrap.cc:654] T f9ae8fac693d4d68b6741f2509e1265e P 525a093ea5bf40dc9bbf9dc39895d670: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:58.601750 10678 raft_consensus.cc:348] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:58.602041 10678 raft_consensus.cc:374] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:58.602146 10678 raft_consensus.cc:729] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Initialized, Role: FOLLOWER
I20240502 21:28:58.602455 10678 consensus_queue.cc:260] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:58.603053 10683 tablet_bootstrap.cc:492] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58: No bootstrap required, opened a new log
I20240502 21:28:58.603161 10678 ts_tablet_manager.cc:1398] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:28:58.603475 10678 tablet_bootstrap.cc:492] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap starting.
I20240502 21:28:58.603447 10683 ts_tablet_manager.cc:1367] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.001s
I20240502 21:28:58.604427 10683 raft_consensus.cc:348] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:58.604722 10683 raft_consensus.cc:374] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:58.604815 10683 raft_consensus.cc:729] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Initialized, Role: FOLLOWER
I20240502 21:28:58.604910 10678 tablet_bootstrap.cc:654] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:58.605121 10683 consensus_queue.cc:260] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:58.605916 10683 ts_tablet_manager.cc:1398] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 21:28:58.606248 10683 tablet_bootstrap.cc:492] T a7c6b881c9ef4cdca787f065778e526f P 5543ddab5b484642b139b7c848932a58: Bootstrap starting.
I20240502 21:28:58.608458 10683 tablet_bootstrap.cc:654] T a7c6b881c9ef4cdca787f065778e526f P 5543ddab5b484642b139b7c848932a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:58.612340 10680 tablet_bootstrap.cc:492] T f9ae8fac693d4d68b6741f2509e1265e P 525a093ea5bf40dc9bbf9dc39895d670: No bootstrap required, opened a new log
I20240502 21:28:58.612598 10680 ts_tablet_manager.cc:1367] T f9ae8fac693d4d68b6741f2509e1265e P 525a093ea5bf40dc9bbf9dc39895d670: Time spent bootstrapping tablet: real 0.021s	user 0.007s	sys 0.000s
I20240502 21:28:58.613641 10680 raft_consensus.cc:348] T f9ae8fac693d4d68b6741f2509e1265e P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:58.613857 10680 raft_consensus.cc:374] T f9ae8fac693d4d68b6741f2509e1265e P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:58.613919 10680 raft_consensus.cc:729] T f9ae8fac693d4d68b6741f2509e1265e P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Initialized, Role: FOLLOWER
I20240502 21:28:58.614138 10680 consensus_queue.cc:260] T f9ae8fac693d4d68b6741f2509e1265e P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:58.614701 10680 ts_tablet_manager.cc:1398] T f9ae8fac693d4d68b6741f2509e1265e P 525a093ea5bf40dc9bbf9dc39895d670: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 21:28:58.615274 10683 tablet_bootstrap.cc:492] T a7c6b881c9ef4cdca787f065778e526f P 5543ddab5b484642b139b7c848932a58: No bootstrap required, opened a new log
I20240502 21:28:58.615463 10683 ts_tablet_manager.cc:1367] T a7c6b881c9ef4cdca787f065778e526f P 5543ddab5b484642b139b7c848932a58: Time spent bootstrapping tablet: real 0.009s	user 0.007s	sys 0.000s
I20240502 21:28:58.616298 10683 raft_consensus.cc:348] T a7c6b881c9ef4cdca787f065778e526f P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:58.616556 10683 raft_consensus.cc:374] T a7c6b881c9ef4cdca787f065778e526f P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:58.616663 10683 raft_consensus.cc:729] T a7c6b881c9ef4cdca787f065778e526f P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Initialized, Role: FOLLOWER
I20240502 21:28:58.616978 10683 consensus_queue.cc:260] T a7c6b881c9ef4cdca787f065778e526f P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:58.617624 10683 ts_tablet_manager.cc:1398] T a7c6b881c9ef4cdca787f065778e526f P 5543ddab5b484642b139b7c848932a58: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:28:58.618916 10678 tablet_bootstrap.cc:492] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0: No bootstrap required, opened a new log
I20240502 21:28:58.619218 10678 ts_tablet_manager.cc:1367] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent bootstrapping tablet: real 0.016s	user 0.007s	sys 0.003s
I20240502 21:28:58.621312 10678 raft_consensus.cc:348] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:58.621565 10678 raft_consensus.cc:374] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:58.621683 10678 raft_consensus.cc:729] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Initialized, Role: FOLLOWER
I20240502 21:28:58.625012 10678 consensus_queue.cc:260] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:58.626366 10604 raft_consensus.cc:482] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:58.626614 10604 raft_consensus.cc:504] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:58.627499 10604 leader_election.cc:288] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:28:58.628217  9240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ae8fac693d4d68b6741f2509e1265e" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670" is_pre_election: true
I20240502 21:28:58.628456  9240 raft_consensus.cc:2454] T f9ae8fac693d4d68b6741f2509e1265e P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 0.
I20240502 21:28:58.628918  9313 leader_election.cc:302] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525a093ea5bf40dc9bbf9dc39895d670, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:28:58.628890  9107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ae8fac693d4d68b6741f2509e1265e" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5543ddab5b484642b139b7c848932a58" is_pre_election: true
I20240502 21:28:58.629228  9107 raft_consensus.cc:2454] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 0.
I20240502 21:28:58.629287 10604 raft_consensus.cc:2781] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:58.629390 10604 raft_consensus.cc:482] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:58.629506 10604 raft_consensus.cc:3037] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:58.630031 10678 ts_tablet_manager.cc:1398] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent starting tablet: real 0.011s	user 0.002s	sys 0.000s
I20240502 21:28:58.632431 10604 raft_consensus.cc:504] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:58.633713  9240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ae8fac693d4d68b6741f2509e1265e" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670"
I20240502 21:28:58.633976  9240 raft_consensus.cc:3037] T f9ae8fac693d4d68b6741f2509e1265e P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:58.635392  9107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ae8fac693d4d68b6741f2509e1265e" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5543ddab5b484642b139b7c848932a58"
I20240502 21:28:58.635659  9107 raft_consensus.cc:3037] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:58.635782 10604 leader_election.cc:288] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 election: Requested vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:28:58.638783  9107 raft_consensus.cc:2454] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 1.
I20240502 21:28:58.639592  9310 leader_election.cc:302] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5543ddab5b484642b139b7c848932a58, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:28:58.640147 10604 raft_consensus.cc:2781] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:58.640403 10604 raft_consensus.cc:686] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Becoming Leader. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Running, Role: LEADER
I20240502 21:28:58.640802 10604 consensus_queue.cc:237] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:28:58.641896  9240 raft_consensus.cc:2454] T f9ae8fac693d4d68b6741f2509e1265e P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 1.
I20240502 21:28:58.646639  8954 catalog_manager.cc:5484] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67). New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } }
I20240502 21:28:58.683832 10604 raft_consensus.cc:482] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:58.684286 10604 raft_consensus.cc:504] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:58.685240 10604 leader_election.cc:288] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
I20240502 21:28:58.686230  9107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be34ec89876d42b69efe1878c0d9bc19" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5543ddab5b484642b139b7c848932a58" is_pre_election: true
I20240502 21:28:58.686293  9508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be34ec89876d42b69efe1878c0d9bc19" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6" is_pre_election: true
I20240502 21:28:58.686586  9107 raft_consensus.cc:2454] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 0.
I20240502 21:28:58.686615  9508 raft_consensus.cc:2454] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 0.
I20240502 21:28:58.687229  9310 leader_election.cc:302] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5543ddab5b484642b139b7c848932a58, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:28:58.687731 10604 raft_consensus.cc:2781] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:58.687885 10604 raft_consensus.cc:482] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:58.687985 10604 raft_consensus.cc:3037] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:58.690412 10604 raft_consensus.cc:504] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:58.691099 10604 leader_election.cc:288] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 election: Requested vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
I20240502 21:28:58.691807  9107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be34ec89876d42b69efe1878c0d9bc19" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5543ddab5b484642b139b7c848932a58"
I20240502 21:28:58.691882  9508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be34ec89876d42b69efe1878c0d9bc19" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6"
I20240502 21:28:58.692075  9107 raft_consensus.cc:3037] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:58.692116  9508 raft_consensus.cc:3037] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:58.694181  9107 raft_consensus.cc:2454] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 1.
I20240502 21:28:58.694192  9508 raft_consensus.cc:2454] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 1.
I20240502 21:28:58.694799  9310 leader_election.cc:302] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:28:58.695401 10604 raft_consensus.cc:2781] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:58.695708 10604 raft_consensus.cc:686] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Becoming Leader. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Running, Role: LEADER
I20240502 21:28:58.696156 10604 consensus_queue.cc:237] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:28:58.700943  8954 catalog_manager.cc:5484] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67). New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } }
I20240502 21:28:58.807332 10604 raft_consensus.cc:482] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:58.807673 10604 raft_consensus.cc:504] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:58.808488 10604 leader_election.cc:288] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:28:58.809403  9107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c6b881c9ef4cdca787f065778e526f" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5543ddab5b484642b139b7c848932a58" is_pre_election: true
I20240502 21:28:58.809367  9240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c6b881c9ef4cdca787f065778e526f" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670" is_pre_election: true
I20240502 21:28:58.809680  9107 raft_consensus.cc:2454] T a7c6b881c9ef4cdca787f065778e526f P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 0.
I20240502 21:28:58.809698  9240 raft_consensus.cc:2454] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 0.
I20240502 21:28:58.810562  9313 leader_election.cc:302] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525a093ea5bf40dc9bbf9dc39895d670, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:28:58.811034 10604 raft_consensus.cc:2781] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:58.811241 10604 raft_consensus.cc:482] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:58.811393 10604 raft_consensus.cc:3037] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:58.813881 10604 raft_consensus.cc:504] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:58.814754 10604 leader_election.cc:288] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 election: Requested vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:28:58.815078  9241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c6b881c9ef4cdca787f065778e526f" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670"
I20240502 21:28:58.815375  9241 raft_consensus.cc:3037] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:58.815482  9107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c6b881c9ef4cdca787f065778e526f" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5543ddab5b484642b139b7c848932a58"
I20240502 21:28:58.815737  9107 raft_consensus.cc:3037] T a7c6b881c9ef4cdca787f065778e526f P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:58.818616  9241 raft_consensus.cc:2454] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 1.
I20240502 21:28:58.818709  9107 raft_consensus.cc:2454] T a7c6b881c9ef4cdca787f065778e526f P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 1.
I20240502 21:28:58.819792  9310 leader_election.cc:302] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 525a093ea5bf40dc9bbf9dc39895d670, 5543ddab5b484642b139b7c848932a58, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:28:58.820549 10604 raft_consensus.cc:2781] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:58.820820 10604 raft_consensus.cc:686] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Becoming Leader. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Running, Role: LEADER
I20240502 21:28:58.821197 10604 consensus_queue.cc:237] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:58.825328  8954 catalog_manager.cc:5484] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67). New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } }
I20240502 21:28:59.069782 10604 consensus_queue.cc:1035] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:59.094408 10604 consensus_queue.cc:1035] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:59.205318 10604 consensus_queue.cc:1035] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:59.211288 10604 consensus_queue.cc:1035] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:59.251942 10604 consensus_queue.cc:1035] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:59.259086 10604 consensus_queue.cc:1035] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:28:59.348124 10694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:59.348567 10694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:59.358843 10694 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
W20240502 21:28:59.603885  9313 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.66:39071, user_credentials={real_user=slave}} blocked reactor thread for 67375.2us

INSERT report
    rows total: 2000
    time total: 206.13 ms
  time per row: 0.103065 ms
I20240502 21:28:59.759932  8956 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54676:
name: "rebalance_test_table_7"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 21:28:59.773881  9490 tablet_service.cc:1467] Processing CreateTablet for tablet 94d2cdf8ae6941668b5f680c558656e8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ef412fa8fe9249729cb99879a5c8343a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.774549  9490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94d2cdf8ae6941668b5f680c558656e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.775164  9357 tablet_service.cc:1467] Processing CreateTablet for tablet 79dedaa327b6479e8325f91481c1688e (DEFAULT_TABLE table=rebalance_test_table_7 [id=ef412fa8fe9249729cb99879a5c8343a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.775089  9489 tablet_service.cc:1467] Processing CreateTablet for tablet 79dedaa327b6479e8325f91481c1688e (DEFAULT_TABLE table=rebalance_test_table_7 [id=ef412fa8fe9249729cb99879a5c8343a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.775667  9357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79dedaa327b6479e8325f91481c1688e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.775810  9489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79dedaa327b6479e8325f91481c1688e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.779343  9219 tablet_service.cc:1467] Processing CreateTablet for tablet 87f35aff2b104c01a9d9d48a7e9c9b23 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ef412fa8fe9249729cb99879a5c8343a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.779837  9219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87f35aff2b104c01a9d9d48a7e9c9b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.780552  9491 tablet_service.cc:1467] Processing CreateTablet for tablet 87f35aff2b104c01a9d9d48a7e9c9b23 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ef412fa8fe9249729cb99879a5c8343a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.781004  9491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87f35aff2b104c01a9d9d48a7e9c9b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.783596  9220 tablet_service.cc:1467] Processing CreateTablet for tablet 94d2cdf8ae6941668b5f680c558656e8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ef412fa8fe9249729cb99879a5c8343a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.784018  9220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94d2cdf8ae6941668b5f680c558656e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.784258  9356 tablet_service.cc:1467] Processing CreateTablet for tablet 87f35aff2b104c01a9d9d48a7e9c9b23 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ef412fa8fe9249729cb99879a5c8343a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.784713  9356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87f35aff2b104c01a9d9d48a7e9c9b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.788069  9087 tablet_service.cc:1467] Processing CreateTablet for tablet 94d2cdf8ae6941668b5f680c558656e8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ef412fa8fe9249729cb99879a5c8343a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.788964  9087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94d2cdf8ae6941668b5f680c558656e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.788125  9086 tablet_service.cc:1467] Processing CreateTablet for tablet 79dedaa327b6479e8325f91481c1688e (DEFAULT_TABLE table=rebalance_test_table_7 [id=ef412fa8fe9249729cb99879a5c8343a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.795902  9086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79dedaa327b6479e8325f91481c1688e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.798595 10759 tablet_bootstrap.cc:492] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap starting.
I20240502 21:28:59.801666 10757 tablet_bootstrap.cc:492] T 94d2cdf8ae6941668b5f680c558656e8 P 19f162e164854f2da623a4bb8f9e6bf6: Bootstrap starting.
I20240502 21:28:59.801705 10758 tablet_bootstrap.cc:492] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap starting.
I20240502 21:28:59.803862 10758 tablet_bootstrap.cc:654] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.803862 10757 tablet_bootstrap.cc:654] T 94d2cdf8ae6941668b5f680c558656e8 P 19f162e164854f2da623a4bb8f9e6bf6: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.805435 10760 tablet_bootstrap.cc:492] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58: Bootstrap starting.
I20240502 21:28:59.805881 10759 tablet_bootstrap.cc:654] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.808696 10760 tablet_bootstrap.cc:654] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.812891 10758 tablet_bootstrap.cc:492] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0: No bootstrap required, opened a new log
I20240502 21:28:59.813215 10758 ts_tablet_manager.cc:1367] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent bootstrapping tablet: real 0.012s	user 0.007s	sys 0.004s
I20240502 21:28:59.814194 10758 raft_consensus.cc:348] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:59.814770 10758 raft_consensus.cc:374] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.814952 10758 raft_consensus.cc:729] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.815373 10758 consensus_queue.cc:260] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:59.816203 10758 ts_tablet_manager.cc:1398] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 21:28:59.816715 10758 tablet_bootstrap.cc:492] T 79dedaa327b6479e8325f91481c1688e P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap starting.
I20240502 21:28:59.818667 10758 tablet_bootstrap.cc:654] T 79dedaa327b6479e8325f91481c1688e P f2fdd596a8b44bbc848fe40f9f07fbe0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.820194 10759 tablet_bootstrap.cc:492] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670: No bootstrap required, opened a new log
I20240502 21:28:59.821460 10759 ts_tablet_manager.cc:1367] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670: Time spent bootstrapping tablet: real 0.023s	user 0.004s	sys 0.013s
I20240502 21:28:59.822451 10759 raft_consensus.cc:348] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:59.822778 10759 raft_consensus.cc:374] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.822937 10759 raft_consensus.cc:729] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.823316 10759 consensus_queue.cc:260] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:59.824865 10759 ts_tablet_manager.cc:1398] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 21:28:59.825248 10757 tablet_bootstrap.cc:492] T 94d2cdf8ae6941668b5f680c558656e8 P 19f162e164854f2da623a4bb8f9e6bf6: No bootstrap required, opened a new log
I20240502 21:28:59.825349 10759 tablet_bootstrap.cc:492] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap starting.
I20240502 21:28:59.825583 10757 ts_tablet_manager.cc:1367] T 94d2cdf8ae6941668b5f680c558656e8 P 19f162e164854f2da623a4bb8f9e6bf6: Time spent bootstrapping tablet: real 0.024s	user 0.008s	sys 0.000s
I20240502 21:28:59.826860 10757 raft_consensus.cc:348] T 94d2cdf8ae6941668b5f680c558656e8 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:59.827275 10759 tablet_bootstrap.cc:654] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.827363 10757 raft_consensus.cc:374] T 94d2cdf8ae6941668b5f680c558656e8 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.827561 10757 raft_consensus.cc:729] T 94d2cdf8ae6941668b5f680c558656e8 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.828011 10757 consensus_queue.cc:260] T 94d2cdf8ae6941668b5f680c558656e8 P 19f162e164854f2da623a4bb8f9e6bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:59.831602 10760 tablet_bootstrap.cc:492] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58: No bootstrap required, opened a new log
I20240502 21:28:59.831951 10760 ts_tablet_manager.cc:1367] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58: Time spent bootstrapping tablet: real 0.027s	user 0.000s	sys 0.016s
I20240502 21:28:59.833321 10760 raft_consensus.cc:348] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:59.833748 10760 raft_consensus.cc:374] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.833932 10760 raft_consensus.cc:729] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.834663 10757 ts_tablet_manager.cc:1398] T 94d2cdf8ae6941668b5f680c558656e8 P 19f162e164854f2da623a4bb8f9e6bf6: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.000s
I20240502 21:28:59.835208 10760 consensus_queue.cc:260] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:59.835976 10757 tablet_bootstrap.cc:492] T 79dedaa327b6479e8325f91481c1688e P 19f162e164854f2da623a4bb8f9e6bf6: Bootstrap starting.
I20240502 21:28:59.836917 10759 tablet_bootstrap.cc:492] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670: No bootstrap required, opened a new log
I20240502 21:28:59.837178 10759 ts_tablet_manager.cc:1367] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670: Time spent bootstrapping tablet: real 0.012s	user 0.001s	sys 0.009s
I20240502 21:28:59.838164 10759 raft_consensus.cc:348] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:59.838501 10759 raft_consensus.cc:374] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.838644 10759 raft_consensus.cc:729] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.838997 10759 consensus_queue.cc:260] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:59.840591 10759 ts_tablet_manager.cc:1398] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 21:28:59.844794 10760 ts_tablet_manager.cc:1398] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58: Time spent starting tablet: real 0.013s	user 0.003s	sys 0.000s
I20240502 21:28:59.847576 10760 tablet_bootstrap.cc:492] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58: Bootstrap starting.
I20240502 21:28:59.849391 10757 tablet_bootstrap.cc:654] T 79dedaa327b6479e8325f91481c1688e P 19f162e164854f2da623a4bb8f9e6bf6: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.854884 10760 tablet_bootstrap.cc:654] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.856019 10758 tablet_bootstrap.cc:492] T 79dedaa327b6479e8325f91481c1688e P f2fdd596a8b44bbc848fe40f9f07fbe0: No bootstrap required, opened a new log
I20240502 21:28:59.856222 10758 ts_tablet_manager.cc:1367] T 79dedaa327b6479e8325f91481c1688e P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent bootstrapping tablet: real 0.040s	user 0.019s	sys 0.013s
I20240502 21:28:59.856508 10757 tablet_bootstrap.cc:492] T 79dedaa327b6479e8325f91481c1688e P 19f162e164854f2da623a4bb8f9e6bf6: No bootstrap required, opened a new log
I20240502 21:28:59.856766 10757 ts_tablet_manager.cc:1367] T 79dedaa327b6479e8325f91481c1688e P 19f162e164854f2da623a4bb8f9e6bf6: Time spent bootstrapping tablet: real 0.021s	user 0.005s	sys 0.003s
I20240502 21:28:59.857043 10758 raft_consensus.cc:348] T 79dedaa327b6479e8325f91481c1688e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:59.857257 10758 raft_consensus.cc:374] T 79dedaa327b6479e8325f91481c1688e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.857338 10758 raft_consensus.cc:729] T 79dedaa327b6479e8325f91481c1688e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.857581 10729 raft_consensus.cc:482] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:59.857621 10758 consensus_queue.cc:260] T 79dedaa327b6479e8325f91481c1688e P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:59.857755 10729 raft_consensus.cc:504] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:59.857988 10757 raft_consensus.cc:348] T 79dedaa327b6479e8325f91481c1688e P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:59.858243 10758 ts_tablet_manager.cc:1398] T 79dedaa327b6479e8325f91481c1688e P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 21:28:59.858306 10757 raft_consensus.cc:374] T 79dedaa327b6479e8325f91481c1688e P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.858389 10757 raft_consensus.cc:729] T 79dedaa327b6479e8325f91481c1688e P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.858660 10729 leader_election.cc:288] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:28:59.858776 10757 consensus_queue.cc:260] T 79dedaa327b6479e8325f91481c1688e P 19f162e164854f2da623a4bb8f9e6bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:59.859738  9508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dedaa327b6479e8325f91481c1688e" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6" is_pre_election: true
I20240502 21:28:59.860023  9508 raft_consensus.cc:2454] T 79dedaa327b6479e8325f91481c1688e P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 0.
I20240502 21:28:59.860882 10757 ts_tablet_manager.cc:1398] T 79dedaa327b6479e8325f91481c1688e P 19f162e164854f2da623a4bb8f9e6bf6: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 21:28:59.861351 10757 tablet_bootstrap.cc:492] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 19f162e164854f2da623a4bb8f9e6bf6: Bootstrap starting.
I20240502 21:28:59.862283  9377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dedaa327b6479e8325f91481c1688e" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" is_pre_election: true
I20240502 21:28:59.862541  9377 raft_consensus.cc:2454] T 79dedaa327b6479e8325f91481c1688e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 0.
I20240502 21:28:59.863171  9038 leader_election.cc:302] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 5543ddab5b484642b139b7c848932a58; no voters: 
I20240502 21:28:59.863977 10660 raft_consensus.cc:2781] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:59.864125 10660 raft_consensus.cc:482] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:59.864202 10757 tablet_bootstrap.cc:654] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 19f162e164854f2da623a4bb8f9e6bf6: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.864235 10660 raft_consensus.cc:3037] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.867041 10760 tablet_bootstrap.cc:492] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58: No bootstrap required, opened a new log
I20240502 21:28:59.867041 10757 tablet_bootstrap.cc:492] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 19f162e164854f2da623a4bb8f9e6bf6: No bootstrap required, opened a new log
I20240502 21:28:59.867247 10757 ts_tablet_manager.cc:1367] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 19f162e164854f2da623a4bb8f9e6bf6: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
I20240502 21:28:59.867271 10760 ts_tablet_manager.cc:1367] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58: Time spent bootstrapping tablet: real 0.020s	user 0.002s	sys 0.012s
I20240502 21:28:59.867421 10714 raft_consensus.cc:482] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:59.867611 10714 raft_consensus.cc:504] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:59.868170 10760 raft_consensus.cc:348] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:59.868185 10757 raft_consensus.cc:348] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:59.868397 10757 raft_consensus.cc:374] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.868417 10760 raft_consensus.cc:374] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.868438 10714 leader_election.cc:288] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:28:59.868491 10760 raft_consensus.cc:729] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.868496 10757 raft_consensus.cc:729] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.868779 10757 consensus_queue.cc:260] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 19f162e164854f2da623a4bb8f9e6bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:59.868835 10760 consensus_queue.cc:260] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:59.869509 10757 ts_tablet_manager.cc:1398] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 19f162e164854f2da623a4bb8f9e6bf6: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:28:59.870141 10760 ts_tablet_manager.cc:1398] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 21:28:59.870208  9106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d2cdf8ae6941668b5f680c558656e8" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5543ddab5b484642b139b7c848932a58" is_pre_election: true
I20240502 21:28:59.870473  9106 raft_consensus.cc:2454] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 0.
I20240502 21:28:59.870975  9172 leader_election.cc:302] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525a093ea5bf40dc9bbf9dc39895d670, 5543ddab5b484642b139b7c848932a58; no voters: 
I20240502 21:28:59.871296  9508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d2cdf8ae6941668b5f680c558656e8" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6" is_pre_election: true
I20240502 21:28:59.871562  9508 raft_consensus.cc:2454] T 94d2cdf8ae6941668b5f680c558656e8 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 0.
I20240502 21:28:59.871816 10403 raft_consensus.cc:2781] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:59.871914 10403 raft_consensus.cc:482] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:59.872007 10403 raft_consensus.cc:3037] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.873133 10660 raft_consensus.cc:504] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:59.875986 10660 leader_election.cc:288] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 election: Requested vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:28:59.876273  9511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dedaa327b6479e8325f91481c1688e" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6"
I20240502 21:28:59.876569  9511 raft_consensus.cc:3037] T 79dedaa327b6479e8325f91481c1688e P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.877681  9377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dedaa327b6479e8325f91481c1688e" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
I20240502 21:28:59.877934  9377 raft_consensus.cc:3037] T 79dedaa327b6479e8325f91481c1688e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.880016  9511 raft_consensus.cc:2454] T 79dedaa327b6479e8325f91481c1688e P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 1.
I20240502 21:28:59.880952  9377 raft_consensus.cc:2454] T 79dedaa327b6479e8325f91481c1688e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 1.
I20240502 21:28:59.881749  9040 leader_election.cc:302] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5543ddab5b484642b139b7c848932a58, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:28:59.882328 10660 raft_consensus.cc:2781] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:59.882695 10660 raft_consensus.cc:686] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Becoming Leader. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Running, Role: LEADER
I20240502 21:28:59.883092 10660 consensus_queue.cc:237] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:28:59.884189 10403 raft_consensus.cc:504] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:59.884388 10714 raft_consensus.cc:482] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:59.884625 10714 raft_consensus.cc:504] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:59.885326 10714 leader_election.cc:288] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
I20240502 21:28:59.887187  9106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d2cdf8ae6941668b5f680c558656e8" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5543ddab5b484642b139b7c848932a58"
I20240502 21:28:59.887434  9106 raft_consensus.cc:3037] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.887392  9509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f35aff2b104c01a9d9d48a7e9c9b23" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6" is_pre_election: true
I20240502 21:28:59.887652  9509 raft_consensus.cc:2454] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 0.
I20240502 21:28:59.889003  9172 leader_election.cc:302] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 525a093ea5bf40dc9bbf9dc39895d670; no voters: 
I20240502 21:28:59.889802  9106 raft_consensus.cc:2454] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 1.
I20240502 21:28:59.889823  9508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d2cdf8ae6941668b5f680c558656e8" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6"
I20240502 21:28:59.890074  9508 raft_consensus.cc:3037] T 94d2cdf8ae6941668b5f680c558656e8 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.891088  9172 leader_election.cc:302] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525a093ea5bf40dc9bbf9dc39895d670, 5543ddab5b484642b139b7c848932a58; no voters: 
I20240502 21:28:59.891832 10714 raft_consensus.cc:2781] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:59.891963 10714 raft_consensus.cc:482] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:59.892112 10714 raft_consensus.cc:3037] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.893097  9508 raft_consensus.cc:2454] T 94d2cdf8ae6941668b5f680c558656e8 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 1.
I20240502 21:28:59.893290 10771 raft_consensus.cc:2781] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:59.893599  9377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f35aff2b104c01a9d9d48a7e9c9b23" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" is_pre_election: true
I20240502 21:28:59.895577 10714 raft_consensus.cc:504] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:59.896435  9377 raft_consensus.cc:2454] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 0.
I20240502 21:28:59.885319 10403 leader_election.cc:288] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 election: Requested vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:28:59.897437 10771 raft_consensus.cc:686] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Becoming Leader. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Running, Role: LEADER
I20240502 21:28:59.897387  8956 catalog_manager.cc:5484] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5543ddab5b484642b139b7c848932a58 (127.27.196.65). New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } }
I20240502 21:28:59.897835 10771 consensus_queue.cc:237] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:59.898111 10714 leader_election.cc:288] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 election: Requested vote from peers f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
I20240502 21:28:59.898494  9377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f35aff2b104c01a9d9d48a7e9c9b23" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
I20240502 21:28:59.898739  9377 raft_consensus.cc:3037] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.899060  9508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f35aff2b104c01a9d9d48a7e9c9b23" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6"
I20240502 21:28:59.899312  9508 raft_consensus.cc:3037] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.902110  9377 raft_consensus.cc:2454] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 1.
I20240502 21:28:59.902604  9174 leader_election.cc:302] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525a093ea5bf40dc9bbf9dc39895d670, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:28:59.902895 10771 raft_consensus.cc:2781] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:59.903082 10771 raft_consensus.cc:686] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Becoming Leader. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Running, Role: LEADER
I20240502 21:28:59.903164  8956 catalog_manager.cc:5484] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } }
I20240502 21:28:59.903379 10771 consensus_queue.cc:237] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:28:59.903875  9508 raft_consensus.cc:2454] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 1.
I20240502 21:28:59.910207  8956 catalog_manager.cc:5484] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } }
I20240502 21:29:00.277208 10771 consensus_queue.cc:1035] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:00.286262 10714 consensus_queue.cc:1035] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:00.321769 10714 consensus_queue.cc:1035] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:00.330502 10771 consensus_queue.cc:1035] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:00.359251 10729 consensus_queue.cc:1035] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:00.364360 10729 consensus_queue.cc:1035] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:29:00.408154 10775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:00.408540 10775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:00.417013 10775 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'

INSERT report
    rows total: 2000
    time total: 311.833 ms
  time per row: 0.155916 ms
I20240502 21:29:00.897032  8958 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54676:
name: "rebalance_test_table_8"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 21:29:00.907649  9220 tablet_service.cc:1467] Processing CreateTablet for tablet c56ac926025e47218e9df4088184a278 (DEFAULT_TABLE table=rebalance_test_table_8 [id=5802da0404e74a07bca290cdc59b279b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:00.907793  9086 tablet_service.cc:1467] Processing CreateTablet for tablet c56ac926025e47218e9df4088184a278 (DEFAULT_TABLE table=rebalance_test_table_8 [id=5802da0404e74a07bca290cdc59b279b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:00.908233  9220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c56ac926025e47218e9df4088184a278. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:00.908480  9086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c56ac926025e47218e9df4088184a278. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:00.910105  9491 tablet_service.cc:1467] Processing CreateTablet for tablet a2f3edaf1c874cefb820f027f90d1185 (DEFAULT_TABLE table=rebalance_test_table_8 [id=5802da0404e74a07bca290cdc59b279b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:00.910758  9491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2f3edaf1c874cefb820f027f90d1185. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:00.910953  9087 tablet_service.cc:1467] Processing CreateTablet for tablet e46065d94e874293a836bc74241b8e06 (DEFAULT_TABLE table=rebalance_test_table_8 [id=5802da0404e74a07bca290cdc59b279b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:00.911568  9087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e46065d94e874293a836bc74241b8e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:00.914341  9357 tablet_service.cc:1467] Processing CreateTablet for tablet a2f3edaf1c874cefb820f027f90d1185 (DEFAULT_TABLE table=rebalance_test_table_8 [id=5802da0404e74a07bca290cdc59b279b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:00.915072  9357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2f3edaf1c874cefb820f027f90d1185. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:00.915656  9356 tablet_service.cc:1467] Processing CreateTablet for tablet e46065d94e874293a836bc74241b8e06 (DEFAULT_TABLE table=rebalance_test_table_8 [id=5802da0404e74a07bca290cdc59b279b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:00.916102  9356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e46065d94e874293a836bc74241b8e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:00.919703  9085 tablet_service.cc:1467] Processing CreateTablet for tablet a2f3edaf1c874cefb820f027f90d1185 (DEFAULT_TABLE table=rebalance_test_table_8 [id=5802da0404e74a07bca290cdc59b279b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:00.919895  9490 tablet_service.cc:1467] Processing CreateTablet for tablet c56ac926025e47218e9df4088184a278 (DEFAULT_TABLE table=rebalance_test_table_8 [id=5802da0404e74a07bca290cdc59b279b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:00.920244  9085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2f3edaf1c874cefb820f027f90d1185. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:00.920359  9490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c56ac926025e47218e9df4088184a278. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:00.924217  9219 tablet_service.cc:1467] Processing CreateTablet for tablet e46065d94e874293a836bc74241b8e06 (DEFAULT_TABLE table=rebalance_test_table_8 [id=5802da0404e74a07bca290cdc59b279b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:00.924945  9219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e46065d94e874293a836bc74241b8e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:00.937341 10842 tablet_bootstrap.cc:492] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6: Bootstrap starting.
I20240502 21:29:00.937561 10841 tablet_bootstrap.cc:492] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58: Bootstrap starting.
I20240502 21:29:00.939934 10842 tablet_bootstrap.cc:654] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:00.941447 10841 tablet_bootstrap.cc:654] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:00.949508 10842 tablet_bootstrap.cc:492] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6: No bootstrap required, opened a new log
I20240502 21:29:00.950001 10842 ts_tablet_manager.cc:1367] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6: Time spent bootstrapping tablet: real 0.013s	user 0.007s	sys 0.004s
I20240502 21:29:00.951507 10842 raft_consensus.cc:348] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:29:00.951961 10842 raft_consensus.cc:374] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:00.952155 10842 raft_consensus.cc:729] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Initialized, Role: FOLLOWER
I20240502 21:29:00.952531 10842 consensus_queue.cc:260] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:29:00.954061 10842 ts_tablet_manager.cc:1398] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 21:29:00.954468 10842 tablet_bootstrap.cc:492] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6: Bootstrap starting.
I20240502 21:29:00.956673 10842 tablet_bootstrap.cc:654] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:00.964293 10841 tablet_bootstrap.cc:492] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58: No bootstrap required, opened a new log
I20240502 21:29:00.964555 10841 ts_tablet_manager.cc:1367] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58: Time spent bootstrapping tablet: real 0.027s	user 0.014s	sys 0.007s
I20240502 21:29:00.965595 10841 raft_consensus.cc:348] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:00.965971 10841 raft_consensus.cc:374] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:00.966017 10844 tablet_bootstrap.cc:492] T e46065d94e874293a836bc74241b8e06 P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap starting.
I20240502 21:29:00.966106 10841 raft_consensus.cc:729] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Initialized, Role: FOLLOWER
I20240502 21:29:00.966437 10841 consensus_queue.cc:260] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:00.968191 10841 ts_tablet_manager.cc:1398] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240502 21:29:00.968533 10844 tablet_bootstrap.cc:654] T e46065d94e874293a836bc74241b8e06 P f2fdd596a8b44bbc848fe40f9f07fbe0: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:00.968580 10841 tablet_bootstrap.cc:492] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58: Bootstrap starting.
I20240502 21:29:00.970451 10842 tablet_bootstrap.cc:492] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6: No bootstrap required, opened a new log
I20240502 21:29:00.970667 10842 ts_tablet_manager.cc:1367] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6: Time spent bootstrapping tablet: real 0.016s	user 0.006s	sys 0.004s
I20240502 21:29:00.971613 10841 tablet_bootstrap.cc:654] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:00.972057 10840 tablet_bootstrap.cc:492] T c56ac926025e47218e9df4088184a278 P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap starting.
I20240502 21:29:00.974586 10842 raft_consensus.cc:348] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:00.974856 10842 raft_consensus.cc:374] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:00.974957 10842 raft_consensus.cc:729] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Initialized, Role: FOLLOWER
I20240502 21:29:00.975517 10842 consensus_queue.cc:260] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:00.976656 10842 ts_tablet_manager.cc:1398] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20240502 21:29:00.978525 10840 tablet_bootstrap.cc:654] T c56ac926025e47218e9df4088184a278 P 525a093ea5bf40dc9bbf9dc39895d670: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:00.982834 10844 tablet_bootstrap.cc:492] T e46065d94e874293a836bc74241b8e06 P f2fdd596a8b44bbc848fe40f9f07fbe0: No bootstrap required, opened a new log
I20240502 21:29:00.983139 10844 ts_tablet_manager.cc:1367] T e46065d94e874293a836bc74241b8e06 P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent bootstrapping tablet: real 0.017s	user 0.007s	sys 0.000s
I20240502 21:29:00.984148 10844 raft_consensus.cc:348] T e46065d94e874293a836bc74241b8e06 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:00.984328 10844 raft_consensus.cc:374] T e46065d94e874293a836bc74241b8e06 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:00.984400 10844 raft_consensus.cc:729] T e46065d94e874293a836bc74241b8e06 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Initialized, Role: FOLLOWER
I20240502 21:29:00.984658 10844 consensus_queue.cc:260] T e46065d94e874293a836bc74241b8e06 P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:00.985263 10844 ts_tablet_manager.cc:1398] T e46065d94e874293a836bc74241b8e06 P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 21:29:00.985500 10841 tablet_bootstrap.cc:492] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58: No bootstrap required, opened a new log
I20240502 21:29:00.985682 10841 ts_tablet_manager.cc:1367] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58: Time spent bootstrapping tablet: real 0.017s	user 0.005s	sys 0.001s
I20240502 21:29:00.986357 10841 raft_consensus.cc:348] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:29:00.986552 10841 raft_consensus.cc:374] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:00.986632 10841 raft_consensus.cc:729] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Initialized, Role: FOLLOWER
I20240502 21:29:00.986891 10841 consensus_queue.cc:260] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:29:00.987404 10844 tablet_bootstrap.cc:492] T a2f3edaf1c874cefb820f027f90d1185 P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap starting.
I20240502 21:29:00.988075 10841 ts_tablet_manager.cc:1398] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:29:00.988396 10841 tablet_bootstrap.cc:492] T a2f3edaf1c874cefb820f027f90d1185 P 5543ddab5b484642b139b7c848932a58: Bootstrap starting.
I20240502 21:29:00.989182 10844 tablet_bootstrap.cc:654] T a2f3edaf1c874cefb820f027f90d1185 P f2fdd596a8b44bbc848fe40f9f07fbe0: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:00.990162 10841 tablet_bootstrap.cc:654] T a2f3edaf1c874cefb820f027f90d1185 P 5543ddab5b484642b139b7c848932a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:00.993268 10840 tablet_bootstrap.cc:492] T c56ac926025e47218e9df4088184a278 P 525a093ea5bf40dc9bbf9dc39895d670: No bootstrap required, opened a new log
I20240502 21:29:00.993445 10840 ts_tablet_manager.cc:1367] T c56ac926025e47218e9df4088184a278 P 525a093ea5bf40dc9bbf9dc39895d670: Time spent bootstrapping tablet: real 0.022s	user 0.003s	sys 0.004s
I20240502 21:29:00.994377 10840 raft_consensus.cc:348] T c56ac926025e47218e9df4088184a278 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:29:00.996263 10840 raft_consensus.cc:374] T c56ac926025e47218e9df4088184a278 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:00.996377 10840 raft_consensus.cc:729] T c56ac926025e47218e9df4088184a278 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Initialized, Role: FOLLOWER
I20240502 21:29:00.996616 10840 consensus_queue.cc:260] T c56ac926025e47218e9df4088184a278 P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:29:00.997341 10840 ts_tablet_manager.cc:1398] T c56ac926025e47218e9df4088184a278 P 525a093ea5bf40dc9bbf9dc39895d670: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.003s
I20240502 21:29:00.998553 10844 tablet_bootstrap.cc:492] T a2f3edaf1c874cefb820f027f90d1185 P f2fdd596a8b44bbc848fe40f9f07fbe0: No bootstrap required, opened a new log
I20240502 21:29:00.998939 10840 tablet_bootstrap.cc:492] T e46065d94e874293a836bc74241b8e06 P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap starting.
I20240502 21:29:00.999262 10844 ts_tablet_manager.cc:1367] T a2f3edaf1c874cefb820f027f90d1185 P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent bootstrapping tablet: real 0.012s	user 0.001s	sys 0.003s
I20240502 21:29:01.000566 10844 raft_consensus.cc:348] T a2f3edaf1c874cefb820f027f90d1185 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:01.001703 10841 tablet_bootstrap.cc:492] T a2f3edaf1c874cefb820f027f90d1185 P 5543ddab5b484642b139b7c848932a58: No bootstrap required, opened a new log
I20240502 21:29:01.001900 10841 ts_tablet_manager.cc:1367] T a2f3edaf1c874cefb820f027f90d1185 P 5543ddab5b484642b139b7c848932a58: Time spent bootstrapping tablet: real 0.014s	user 0.001s	sys 0.008s
I20240502 21:29:01.002641 10840 tablet_bootstrap.cc:654] T e46065d94e874293a836bc74241b8e06 P 525a093ea5bf40dc9bbf9dc39895d670: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:01.003046 10841 raft_consensus.cc:348] T a2f3edaf1c874cefb820f027f90d1185 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:01.003257 10841 raft_consensus.cc:374] T a2f3edaf1c874cefb820f027f90d1185 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:01.003330 10841 raft_consensus.cc:729] T a2f3edaf1c874cefb820f027f90d1185 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Initialized, Role: FOLLOWER
I20240502 21:29:01.003563 10844 raft_consensus.cc:374] T a2f3edaf1c874cefb820f027f90d1185 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:01.003532 10841 consensus_queue.cc:260] T a2f3edaf1c874cefb820f027f90d1185 P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:01.003698 10844 raft_consensus.cc:729] T a2f3edaf1c874cefb820f027f90d1185 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Initialized, Role: FOLLOWER
I20240502 21:29:01.003988 10844 consensus_queue.cc:260] T a2f3edaf1c874cefb820f027f90d1185 P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:01.004545 10844 ts_tablet_manager.cc:1398] T a2f3edaf1c874cefb820f027f90d1185 P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.000s
I20240502 21:29:01.005095 10841 ts_tablet_manager.cc:1398] T a2f3edaf1c874cefb820f027f90d1185 P 5543ddab5b484642b139b7c848932a58: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 21:29:01.006018 10840 tablet_bootstrap.cc:492] T e46065d94e874293a836bc74241b8e06 P 525a093ea5bf40dc9bbf9dc39895d670: No bootstrap required, opened a new log
I20240502 21:29:01.006251 10840 ts_tablet_manager.cc:1367] T e46065d94e874293a836bc74241b8e06 P 525a093ea5bf40dc9bbf9dc39895d670: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.003s
I20240502 21:29:01.007848 10840 raft_consensus.cc:348] T e46065d94e874293a836bc74241b8e06 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:01.008141 10840 raft_consensus.cc:374] T e46065d94e874293a836bc74241b8e06 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:01.008234 10840 raft_consensus.cc:729] T e46065d94e874293a836bc74241b8e06 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Initialized, Role: FOLLOWER
I20240502 21:29:01.008503 10840 consensus_queue.cc:260] T e46065d94e874293a836bc74241b8e06 P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:01.010008 10840 ts_tablet_manager.cc:1398] T e46065d94e874293a836bc74241b8e06 P 525a093ea5bf40dc9bbf9dc39895d670: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.003s
I20240502 21:29:01.053350 10828 raft_consensus.cc:482] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:01.053706 10828 raft_consensus.cc:504] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:01.054672 10828 leader_election.cc:288] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:29:01.055649  9241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46065d94e874293a836bc74241b8e06" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670" is_pre_election: true
I20240502 21:29:01.055692  9377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46065d94e874293a836bc74241b8e06" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" is_pre_election: true
I20240502 21:29:01.055984  9241 raft_consensus.cc:2454] T e46065d94e874293a836bc74241b8e06 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 0.
I20240502 21:29:01.056013  9377 raft_consensus.cc:2454] T e46065d94e874293a836bc74241b8e06 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 0.
I20240502 21:29:01.056743  9040 leader_election.cc:302] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 525a093ea5bf40dc9bbf9dc39895d670, 5543ddab5b484642b139b7c848932a58, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:29:01.057209 10828 raft_consensus.cc:2781] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:01.057368 10828 raft_consensus.cc:482] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:01.057511 10828 raft_consensus.cc:3037] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:01.059911 10828 raft_consensus.cc:504] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:01.060609 10828 leader_election.cc:288] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 election: Requested vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:29:01.061709  9377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46065d94e874293a836bc74241b8e06" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
I20240502 21:29:01.061704  9241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46065d94e874293a836bc74241b8e06" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670"
I20240502 21:29:01.062075  9377 raft_consensus.cc:3037] T e46065d94e874293a836bc74241b8e06 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:01.062074  9241 raft_consensus.cc:3037] T e46065d94e874293a836bc74241b8e06 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:01.065642  9241 raft_consensus.cc:2454] T e46065d94e874293a836bc74241b8e06 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 1.
I20240502 21:29:01.065642  9377 raft_consensus.cc:2454] T e46065d94e874293a836bc74241b8e06 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 1.
I20240502 21:29:01.066476  9040 leader_election.cc:302] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 525a093ea5bf40dc9bbf9dc39895d670, 5543ddab5b484642b139b7c848932a58, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:29:01.067010 10828 raft_consensus.cc:2781] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:01.067297 10828 raft_consensus.cc:686] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Becoming Leader. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Running, Role: LEADER
I20240502 21:29:01.067698 10828 consensus_queue.cc:237] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:01.072131  8954 catalog_manager.cc:5484] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5543ddab5b484642b139b7c848932a58 (127.27.196.65). New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } }
I20240502 21:29:01.080348 10623 raft_consensus.cc:482] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:01.080646 10623 raft_consensus.cc:504] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:29:01.081936 10623 leader_election.cc:288] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:29:01.082818  9241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56ac926025e47218e9df4088184a278" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670" is_pre_election: true
I20240502 21:29:01.083247  9241 raft_consensus.cc:2454] T c56ac926025e47218e9df4088184a278 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19f162e164854f2da623a4bb8f9e6bf6 in term 0.
I20240502 21:29:01.083822  9447 leader_election.cc:302] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 525a093ea5bf40dc9bbf9dc39895d670; no voters: 
I20240502 21:29:01.084293 10623 raft_consensus.cc:2781] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:01.084473 10623 raft_consensus.cc:482] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:01.084619 10623 raft_consensus.cc:3037] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:01.086994 10623 raft_consensus.cc:504] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:29:01.088429 10623 leader_election.cc:288] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 1 election: Requested vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:29:01.088726  9241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56ac926025e47218e9df4088184a278" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670"
I20240502 21:29:01.088986  9241 raft_consensus.cc:3037] T c56ac926025e47218e9df4088184a278 P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:01.091163  9108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56ac926025e47218e9df4088184a278" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5543ddab5b484642b139b7c848932a58"
I20240502 21:29:01.091091  9106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56ac926025e47218e9df4088184a278" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5543ddab5b484642b139b7c848932a58" is_pre_election: true
I20240502 21:29:01.091305  9241 raft_consensus.cc:2454] T c56ac926025e47218e9df4088184a278 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f162e164854f2da623a4bb8f9e6bf6 in term 1.
I20240502 21:29:01.091418  9108 raft_consensus.cc:3037] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:01.091792  9447 leader_election.cc:302] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 525a093ea5bf40dc9bbf9dc39895d670; no voters: 
I20240502 21:29:01.092160 10623 raft_consensus.cc:2781] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:01.092629 10623 raft_consensus.cc:686] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 LEADER]: Becoming Leader. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Running, Role: LEADER
I20240502 21:29:01.093050 10623 consensus_queue.cc:237] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:29:01.094321  9108 raft_consensus.cc:2454] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f162e164854f2da623a4bb8f9e6bf6 in term 1.
I20240502 21:29:01.096948  8954 catalog_manager.cc:5484] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68). New cstate: current_term: 1 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } }
I20240502 21:29:01.165190 10623 raft_consensus.cc:482] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:01.165418 10623 raft_consensus.cc:504] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:01.166116 10623 leader_election.cc:288] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:29:01.166779  9377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f3edaf1c874cefb820f027f90d1185" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" is_pre_election: true
I20240502 21:29:01.166893  9108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f3edaf1c874cefb820f027f90d1185" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5543ddab5b484642b139b7c848932a58" is_pre_election: true
I20240502 21:29:01.167064  9377 raft_consensus.cc:2454] T a2f3edaf1c874cefb820f027f90d1185 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19f162e164854f2da623a4bb8f9e6bf6 in term 0.
I20240502 21:29:01.167222  9108 raft_consensus.cc:2454] T a2f3edaf1c874cefb820f027f90d1185 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19f162e164854f2da623a4bb8f9e6bf6 in term 0.
I20240502 21:29:01.167603  9446 leader_election.cc:302] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:29:01.167940 10623 raft_consensus.cc:2781] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:01.168030 10623 raft_consensus.cc:482] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:01.168139 10623 raft_consensus.cc:3037] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:01.170368 10623 raft_consensus.cc:504] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:01.171202 10623 leader_election.cc:288] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 1 election: Requested vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:29:01.171800  9377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f3edaf1c874cefb820f027f90d1185" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
I20240502 21:29:01.172032  9377 raft_consensus.cc:3037] T a2f3edaf1c874cefb820f027f90d1185 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:01.172284  9108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f3edaf1c874cefb820f027f90d1185" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5543ddab5b484642b139b7c848932a58"
I20240502 21:29:01.172569  9108 raft_consensus.cc:3037] T a2f3edaf1c874cefb820f027f90d1185 P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:01.174242  9377 raft_consensus.cc:2454] T a2f3edaf1c874cefb820f027f90d1185 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f162e164854f2da623a4bb8f9e6bf6 in term 1.
I20240502 21:29:01.174744  9446 leader_election.cc:302] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:29:01.174772  9108 raft_consensus.cc:2454] T a2f3edaf1c874cefb820f027f90d1185 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f162e164854f2da623a4bb8f9e6bf6 in term 1.
I20240502 21:29:01.175084 10623 raft_consensus.cc:2781] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:01.175318 10623 raft_consensus.cc:686] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 LEADER]: Becoming Leader. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Running, Role: LEADER
I20240502 21:29:01.175635 10623 consensus_queue.cc:237] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:01.179528  8954 catalog_manager.cc:5484] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68). New cstate: current_term: 1 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } }
I20240502 21:29:01.462143 10667 consensus_queue.cc:1035] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:01.470551 10828 consensus_queue.cc:1035] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:01.596264 10623 consensus_queue.cc:1035] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:01.604220 10856 consensus_queue.cc:1035] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:01.638612 10856 consensus_queue.cc:1035] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:01.645874 10623 consensus_queue.cc:1035] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:29:01.715359 10859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:01.715910 10859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:01.725761 10859 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'

INSERT report
    rows total: 2000
    time total: 255.265 ms
  time per row: 0.127633 ms
I20240502 21:29:02.175491  8954 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54676:
name: "rebalance_test_table_9"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 21:29:02.189922  9220 tablet_service.cc:1467] Processing CreateTablet for tablet 3caa36212ba444aa88d153ff9f90122e (DEFAULT_TABLE table=rebalance_test_table_9 [id=16e8100140034581b2a773dcf740998d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:02.190472  9220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3caa36212ba444aa88d153ff9f90122e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:02.191036  9356 tablet_service.cc:1467] Processing CreateTablet for tablet 3caa36212ba444aa88d153ff9f90122e (DEFAULT_TABLE table=rebalance_test_table_9 [id=16e8100140034581b2a773dcf740998d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:02.191546  9356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3caa36212ba444aa88d153ff9f90122e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:02.191609  9491 tablet_service.cc:1467] Processing CreateTablet for tablet 3caa36212ba444aa88d153ff9f90122e (DEFAULT_TABLE table=rebalance_test_table_9 [id=16e8100140034581b2a773dcf740998d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:02.192044  9491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3caa36212ba444aa88d153ff9f90122e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:02.195315  9219 tablet_service.cc:1467] Processing CreateTablet for tablet 8129d13163f64368b731ba4eaeaaa81c (DEFAULT_TABLE table=rebalance_test_table_9 [id=16e8100140034581b2a773dcf740998d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:02.195518  9490 tablet_service.cc:1467] Processing CreateTablet for tablet 8129d13163f64368b731ba4eaeaaa81c (DEFAULT_TABLE table=rebalance_test_table_9 [id=16e8100140034581b2a773dcf740998d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:02.195811  9219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8129d13163f64368b731ba4eaeaaa81c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:02.195902  9490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8129d13163f64368b731ba4eaeaaa81c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:02.199184  9357 tablet_service.cc:1467] Processing CreateTablet for tablet 8129d13163f64368b731ba4eaeaaa81c (DEFAULT_TABLE table=rebalance_test_table_9 [id=16e8100140034581b2a773dcf740998d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:02.199661  9357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8129d13163f64368b731ba4eaeaaa81c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:02.200691  9085 tablet_service.cc:1467] Processing CreateTablet for tablet 3d1a5050d3f944969a560994a23700ad (DEFAULT_TABLE table=rebalance_test_table_9 [id=16e8100140034581b2a773dcf740998d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:02.201377  9085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1a5050d3f944969a560994a23700ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:02.214040  9221 tablet_service.cc:1467] Processing CreateTablet for tablet 3d1a5050d3f944969a560994a23700ad (DEFAULT_TABLE table=rebalance_test_table_9 [id=16e8100140034581b2a773dcf740998d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:02.214609  9221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1a5050d3f944969a560994a23700ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:02.229302 10926 tablet_bootstrap.cc:492] T 8129d13163f64368b731ba4eaeaaa81c P 19f162e164854f2da623a4bb8f9e6bf6: Bootstrap starting.
I20240502 21:29:02.229831 10928 tablet_bootstrap.cc:492] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap starting.
I20240502 21:29:02.232302 10926 tablet_bootstrap.cc:654] T 8129d13163f64368b731ba4eaeaaa81c P 19f162e164854f2da623a4bb8f9e6bf6: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:02.234853 10928 tablet_bootstrap.cc:654] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:02.242319 10928 tablet_bootstrap.cc:492] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670: No bootstrap required, opened a new log
I20240502 21:29:02.243537 10928 ts_tablet_manager.cc:1367] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670: Time spent bootstrapping tablet: real 0.014s	user 0.006s	sys 0.000s
I20240502 21:29:02.245059 10928 raft_consensus.cc:348] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:02.246974 10928 raft_consensus.cc:374] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:02.247684 10928 raft_consensus.cc:729] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Initialized, Role: FOLLOWER
I20240502 21:29:02.248492 10928 consensus_queue.cc:260] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:02.250219 10928 ts_tablet_manager.cc:1398] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240502 21:29:02.251441 10928 tablet_bootstrap.cc:492] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap starting.
I20240502 21:29:02.256191  9490 tablet_service.cc:1467] Processing CreateTablet for tablet 3d1a5050d3f944969a560994a23700ad (DEFAULT_TABLE table=rebalance_test_table_9 [id=16e8100140034581b2a773dcf740998d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:29:02.256841  9490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1a5050d3f944969a560994a23700ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:02.269451 10928 tablet_bootstrap.cc:654] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:02.269639 10931 tablet_bootstrap.cc:492] T 3d1a5050d3f944969a560994a23700ad P 5543ddab5b484642b139b7c848932a58: Bootstrap starting.
I20240502 21:29:02.273818 10931 tablet_bootstrap.cc:654] T 3d1a5050d3f944969a560994a23700ad P 5543ddab5b484642b139b7c848932a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:02.276942 10928 tablet_bootstrap.cc:492] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670: No bootstrap required, opened a new log
I20240502 21:29:02.277972 10928 ts_tablet_manager.cc:1367] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670: Time spent bootstrapping tablet: real 0.027s	user 0.002s	sys 0.004s
I20240502 21:29:02.279855 10927 tablet_bootstrap.cc:492] T 3caa36212ba444aa88d153ff9f90122e P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap starting.
I20240502 21:29:02.279695 10928 raft_consensus.cc:348] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:02.280496 10928 raft_consensus.cc:374] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:02.280612 10928 raft_consensus.cc:729] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Initialized, Role: FOLLOWER
I20240502 21:29:02.281128 10928 consensus_queue.cc:260] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:02.282186 10928 ts_tablet_manager.cc:1398] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.001s
I20240502 21:29:02.282624 10928 tablet_bootstrap.cc:492] T 3d1a5050d3f944969a560994a23700ad P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap starting.
I20240502 21:29:02.285214 10927 tablet_bootstrap.cc:654] T 3caa36212ba444aa88d153ff9f90122e P f2fdd596a8b44bbc848fe40f9f07fbe0: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:02.285995 10928 tablet_bootstrap.cc:654] T 3d1a5050d3f944969a560994a23700ad P 525a093ea5bf40dc9bbf9dc39895d670: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:02.288062 10926 tablet_bootstrap.cc:492] T 8129d13163f64368b731ba4eaeaaa81c P 19f162e164854f2da623a4bb8f9e6bf6: No bootstrap required, opened a new log
I20240502 21:29:02.288453 10926 ts_tablet_manager.cc:1367] T 8129d13163f64368b731ba4eaeaaa81c P 19f162e164854f2da623a4bb8f9e6bf6: Time spent bootstrapping tablet: real 0.059s	user 0.027s	sys 0.029s
I20240502 21:29:02.289806 10926 raft_consensus.cc:348] T 8129d13163f64368b731ba4eaeaaa81c P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:02.290258 10926 raft_consensus.cc:374] T 8129d13163f64368b731ba4eaeaaa81c P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:02.290436 10926 raft_consensus.cc:729] T 8129d13163f64368b731ba4eaeaaa81c P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Initialized, Role: FOLLOWER
I20240502 21:29:02.290982 10926 consensus_queue.cc:260] T 8129d13163f64368b731ba4eaeaaa81c P 19f162e164854f2da623a4bb8f9e6bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:02.292557 10931 tablet_bootstrap.cc:492] T 3d1a5050d3f944969a560994a23700ad P 5543ddab5b484642b139b7c848932a58: No bootstrap required, opened a new log
I20240502 21:29:02.292749 10931 ts_tablet_manager.cc:1367] T 3d1a5050d3f944969a560994a23700ad P 5543ddab5b484642b139b7c848932a58: Time spent bootstrapping tablet: real 0.023s	user 0.001s	sys 0.004s
I20240502 21:29:02.294032 10926 ts_tablet_manager.cc:1398] T 8129d13163f64368b731ba4eaeaaa81c P 19f162e164854f2da623a4bb8f9e6bf6: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 21:29:02.294114 10931 raft_consensus.cc:348] T 3d1a5050d3f944969a560994a23700ad P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:29:02.294472 10931 raft_consensus.cc:374] T 3d1a5050d3f944969a560994a23700ad P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:02.294605 10926 tablet_bootstrap.cc:492] T 3caa36212ba444aa88d153ff9f90122e P 19f162e164854f2da623a4bb8f9e6bf6: Bootstrap starting.
I20240502 21:29:02.294603 10931 raft_consensus.cc:729] T 3d1a5050d3f944969a560994a23700ad P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Initialized, Role: FOLLOWER
I20240502 21:29:02.298187 10926 tablet_bootstrap.cc:654] T 3caa36212ba444aa88d153ff9f90122e P 19f162e164854f2da623a4bb8f9e6bf6: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:02.299507 10931 consensus_queue.cc:260] T 3d1a5050d3f944969a560994a23700ad P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:29:02.301378 10928 tablet_bootstrap.cc:492] T 3d1a5050d3f944969a560994a23700ad P 525a093ea5bf40dc9bbf9dc39895d670: No bootstrap required, opened a new log
I20240502 21:29:02.301606 10928 ts_tablet_manager.cc:1367] T 3d1a5050d3f944969a560994a23700ad P 525a093ea5bf40dc9bbf9dc39895d670: Time spent bootstrapping tablet: real 0.019s	user 0.004s	sys 0.009s
I20240502 21:29:02.302357 10931 ts_tablet_manager.cc:1398] T 3d1a5050d3f944969a560994a23700ad P 5543ddab5b484642b139b7c848932a58: Time spent starting tablet: real 0.009s	user 0.000s	sys 0.003s
I20240502 21:29:02.302727 10928 raft_consensus.cc:348] T 3d1a5050d3f944969a560994a23700ad P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:29:02.303022 10928 raft_consensus.cc:374] T 3d1a5050d3f944969a560994a23700ad P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:02.303143 10928 raft_consensus.cc:729] T 3d1a5050d3f944969a560994a23700ad P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Initialized, Role: FOLLOWER
I20240502 21:29:02.303467 10928 consensus_queue.cc:260] T 3d1a5050d3f944969a560994a23700ad P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:29:02.305354 10928 ts_tablet_manager.cc:1398] T 3d1a5050d3f944969a560994a23700ad P 525a093ea5bf40dc9bbf9dc39895d670: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.003s
I20240502 21:29:02.306252 10926 tablet_bootstrap.cc:492] T 3caa36212ba444aa88d153ff9f90122e P 19f162e164854f2da623a4bb8f9e6bf6: No bootstrap required, opened a new log
I20240502 21:29:02.306440 10926 ts_tablet_manager.cc:1367] T 3caa36212ba444aa88d153ff9f90122e P 19f162e164854f2da623a4bb8f9e6bf6: Time spent bootstrapping tablet: real 0.012s	user 0.007s	sys 0.000s
I20240502 21:29:02.307441 10926 raft_consensus.cc:348] T 3caa36212ba444aa88d153ff9f90122e P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:02.307763 10926 raft_consensus.cc:374] T 3caa36212ba444aa88d153ff9f90122e P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:02.307834 10926 raft_consensus.cc:729] T 3caa36212ba444aa88d153ff9f90122e P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Initialized, Role: FOLLOWER
I20240502 21:29:02.308149 10926 consensus_queue.cc:260] T 3caa36212ba444aa88d153ff9f90122e P 19f162e164854f2da623a4bb8f9e6bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:02.308846 10926 ts_tablet_manager.cc:1398] T 3caa36212ba444aa88d153ff9f90122e P 19f162e164854f2da623a4bb8f9e6bf6: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:29:02.309162 10926 tablet_bootstrap.cc:492] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6: Bootstrap starting.
I20240502 21:29:02.311878 10926 tablet_bootstrap.cc:654] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:02.312937 10927 tablet_bootstrap.cc:492] T 3caa36212ba444aa88d153ff9f90122e P f2fdd596a8b44bbc848fe40f9f07fbe0: No bootstrap required, opened a new log
I20240502 21:29:02.313164 10927 ts_tablet_manager.cc:1367] T 3caa36212ba444aa88d153ff9f90122e P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent bootstrapping tablet: real 0.033s	user 0.003s	sys 0.002s
I20240502 21:29:02.314339 10927 raft_consensus.cc:348] T 3caa36212ba444aa88d153ff9f90122e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:02.318709 10927 raft_consensus.cc:374] T 3caa36212ba444aa88d153ff9f90122e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:02.318917 10927 raft_consensus.cc:729] T 3caa36212ba444aa88d153ff9f90122e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Initialized, Role: FOLLOWER
I20240502 21:29:02.319279 10927 consensus_queue.cc:260] T 3caa36212ba444aa88d153ff9f90122e P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:02.320195 10926 tablet_bootstrap.cc:492] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6: No bootstrap required, opened a new log
I20240502 21:29:02.320380 10926 ts_tablet_manager.cc:1367] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6: Time spent bootstrapping tablet: real 0.011s	user 0.009s	sys 0.000s
I20240502 21:29:02.321209 10926 raft_consensus.cc:348] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:29:02.321424 10926 raft_consensus.cc:374] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:02.321524 10926 raft_consensus.cc:729] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Initialized, Role: FOLLOWER
I20240502 21:29:02.321796 10926 consensus_queue.cc:260] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:29:02.322428 10926 ts_tablet_manager.cc:1398] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 21:29:02.324896 10927 ts_tablet_manager.cc:1398] T 3caa36212ba444aa88d153ff9f90122e P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent starting tablet: real 0.011s	user 0.002s	sys 0.001s
I20240502 21:29:02.325454 10927 tablet_bootstrap.cc:492] T 8129d13163f64368b731ba4eaeaaa81c P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap starting.
I20240502 21:29:02.330521 10927 tablet_bootstrap.cc:654] T 8129d13163f64368b731ba4eaeaaa81c P f2fdd596a8b44bbc848fe40f9f07fbe0: Neither blocks nor log segments found. Creating new log.
I20240502 21:29:02.335927 10927 tablet_bootstrap.cc:492] T 8129d13163f64368b731ba4eaeaaa81c P f2fdd596a8b44bbc848fe40f9f07fbe0: No bootstrap required, opened a new log
I20240502 21:29:02.336136 10927 ts_tablet_manager.cc:1367] T 8129d13163f64368b731ba4eaeaaa81c P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent bootstrapping tablet: real 0.011s	user 0.004s	sys 0.000s
I20240502 21:29:02.337006 10927 raft_consensus.cc:348] T 8129d13163f64368b731ba4eaeaaa81c P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:02.337258 10927 raft_consensus.cc:374] T 8129d13163f64368b731ba4eaeaaa81c P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:29:02.337330 10927 raft_consensus.cc:729] T 8129d13163f64368b731ba4eaeaaa81c P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Initialized, Role: FOLLOWER
I20240502 21:29:02.337639 10927 consensus_queue.cc:260] T 8129d13163f64368b731ba4eaeaaa81c P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:02.338287 10927 ts_tablet_manager.cc:1398] T 8129d13163f64368b731ba4eaeaaa81c P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 21:29:02.355743 10904 raft_consensus.cc:482] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:02.355995 10904 raft_consensus.cc:504] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:02.356803 10904 leader_election.cc:288] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:29:02.357465  9508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3caa36212ba444aa88d153ff9f90122e" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6" is_pre_election: true
I20240502 21:29:02.357487  9377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3caa36212ba444aa88d153ff9f90122e" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" is_pre_election: true
I20240502 21:29:02.357703  9508 raft_consensus.cc:2454] T 3caa36212ba444aa88d153ff9f90122e P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 0.
I20240502 21:29:02.357792  9377 raft_consensus.cc:2454] T 3caa36212ba444aa88d153ff9f90122e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 0.
I20240502 21:29:02.358122  9172 leader_election.cc:302] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 525a093ea5bf40dc9bbf9dc39895d670; no voters: 
I20240502 21:29:02.358510 10904 raft_consensus.cc:2781] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:02.358610 10904 raft_consensus.cc:482] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:02.358706 10904 raft_consensus.cc:3037] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:02.360986 10904 raft_consensus.cc:504] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:02.362330  9508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3caa36212ba444aa88d153ff9f90122e" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6"
I20240502 21:29:02.362653  9508 raft_consensus.cc:3037] T 3caa36212ba444aa88d153ff9f90122e P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:02.362778 10904 leader_election.cc:288] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 election: Requested vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:29:02.363839  9377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3caa36212ba444aa88d153ff9f90122e" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
I20240502 21:29:02.364171  9377 raft_consensus.cc:3037] T 3caa36212ba444aa88d153ff9f90122e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:02.365146  9508 raft_consensus.cc:2454] T 3caa36212ba444aa88d153ff9f90122e P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 1.
I20240502 21:29:02.366127  9172 leader_election.cc:302] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 525a093ea5bf40dc9bbf9dc39895d670; no voters: 
I20240502 21:29:02.366514 10904 raft_consensus.cc:2781] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:02.366683 10904 raft_consensus.cc:686] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Becoming Leader. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Running, Role: LEADER
I20240502 21:29:02.367048 10904 consensus_queue.cc:237] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:02.371731  8960 catalog_manager.cc:5484] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } }
I20240502 21:29:02.374292  9377 raft_consensus.cc:2454] T 3caa36212ba444aa88d153ff9f90122e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 1.
I20240502 21:29:02.510401 10904 raft_consensus.cc:482] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:02.510756 10904 raft_consensus.cc:504] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:02.511790 10904 leader_election.cc:288] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:29:02.512723  9377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8129d13163f64368b731ba4eaeaaa81c" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" is_pre_election: true
I20240502 21:29:02.512823  9508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8129d13163f64368b731ba4eaeaaa81c" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6" is_pre_election: true
I20240502 21:29:02.513057  9377 raft_consensus.cc:2454] T 8129d13163f64368b731ba4eaeaaa81c P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 0.
I20240502 21:29:02.513155  9508 raft_consensus.cc:2454] T 8129d13163f64368b731ba4eaeaaa81c P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 0.
I20240502 21:29:02.513824  9174 leader_election.cc:302] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525a093ea5bf40dc9bbf9dc39895d670, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:29:02.514482 10904 raft_consensus.cc:2781] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:02.514603 10904 raft_consensus.cc:482] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:02.514693 10904 raft_consensus.cc:3037] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:02.516781 10904 raft_consensus.cc:504] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:02.517730 10904 leader_election.cc:288] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 election: Requested vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:29:02.518349  9377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8129d13163f64368b731ba4eaeaaa81c" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
I20240502 21:29:02.518462  9508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8129d13163f64368b731ba4eaeaaa81c" candidate_uuid: "525a093ea5bf40dc9bbf9dc39895d670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6"
I20240502 21:29:02.518633  9377 raft_consensus.cc:3037] T 8129d13163f64368b731ba4eaeaaa81c P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:02.518798  9508 raft_consensus.cc:3037] T 8129d13163f64368b731ba4eaeaaa81c P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:02.519232 10867 raft_consensus.cc:482] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:29:02.519428 10867 raft_consensus.cc:504] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:29:02.520843 10867 leader_election.cc:288] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:29:02.521205  9106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1a5050d3f944969a560994a23700ad" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5543ddab5b484642b139b7c848932a58" is_pre_election: true
I20240502 21:29:02.521188  9242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1a5050d3f944969a560994a23700ad" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670" is_pre_election: true
I20240502 21:29:02.521467  9106 raft_consensus.cc:2454] T 3d1a5050d3f944969a560994a23700ad P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19f162e164854f2da623a4bb8f9e6bf6 in term 0.
I20240502 21:29:02.521483  9242 raft_consensus.cc:2454] T 3d1a5050d3f944969a560994a23700ad P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19f162e164854f2da623a4bb8f9e6bf6 in term 0.
I20240502 21:29:02.522076  9508 raft_consensus.cc:2454] T 8129d13163f64368b731ba4eaeaaa81c P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 1.
I20240502 21:29:02.522043  9447 leader_election.cc:302] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 525a093ea5bf40dc9bbf9dc39895d670; no voters: 
I20240502 21:29:02.522126  9377 raft_consensus.cc:2454] T 8129d13163f64368b731ba4eaeaaa81c P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525a093ea5bf40dc9bbf9dc39895d670 in term 1.
I20240502 21:29:02.522404 10867 raft_consensus.cc:2781] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:29:02.522545 10867 raft_consensus.cc:482] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:29:02.522651 10867 raft_consensus.cc:3037] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:02.522601  9174 leader_election.cc:302] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525a093ea5bf40dc9bbf9dc39895d670, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:29:02.522919 10904 raft_consensus.cc:2781] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:02.523159 10904 raft_consensus.cc:686] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Becoming Leader. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Running, Role: LEADER
I20240502 21:29:02.523538 10904 consensus_queue.cc:237] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:29:02.525735 10867 raft_consensus.cc:504] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:29:02.526638 10867 leader_election.cc:288] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 1 election: Requested vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:29:02.527806  9242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1a5050d3f944969a560994a23700ad" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670"
I20240502 21:29:02.528262  9242 raft_consensus.cc:3037] T 3d1a5050d3f944969a560994a23700ad P 525a093ea5bf40dc9bbf9dc39895d670 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:02.529727  9106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1a5050d3f944969a560994a23700ad" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5543ddab5b484642b139b7c848932a58"
I20240502 21:29:02.530063  9106 raft_consensus.cc:3037] T 3d1a5050d3f944969a560994a23700ad P 5543ddab5b484642b139b7c848932a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:29:02.530346  8960 catalog_manager.cc:5484] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } }
I20240502 21:29:02.533144  9242 raft_consensus.cc:2454] T 3d1a5050d3f944969a560994a23700ad P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f162e164854f2da623a4bb8f9e6bf6 in term 1.
I20240502 21:29:02.533732  9447 leader_election.cc:302] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 525a093ea5bf40dc9bbf9dc39895d670; no voters: 
I20240502 21:29:02.534112  9106 raft_consensus.cc:2454] T 3d1a5050d3f944969a560994a23700ad P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f162e164854f2da623a4bb8f9e6bf6 in term 1.
I20240502 21:29:02.534198 10856 raft_consensus.cc:2781] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:29:02.535169 10856 raft_consensus.cc:686] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 LEADER]: Becoming Leader. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Running, Role: LEADER
I20240502 21:29:02.535586 10856 consensus_queue.cc:237] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:29:02.539444  8960 catalog_manager.cc:5484] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68). New cstate: current_term: 1 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } }
I20240502 21:29:02.915858 10904 consensus_queue.cc:1035] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:02.920593 10904 consensus_queue.cc:1035] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:02.971732 10904 consensus_queue.cc:1035] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:02.985635 10832 consensus_queue.cc:1035] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:29:03.046639 10944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:03.047057 10944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:03.055999 10944 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240502 21:29:03.128674 10941 consensus_queue.cc:1035] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:03.168344 10975 consensus_queue.cc:1035] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 274.348 ms
  time per row: 0.137174 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      4       1192 libcrypto.so.1.1
-----------------------------------------------------

I20240502 21:29:04.545511 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.69:42049
--local_ip_for_outbound_sockets=127.27.196.69
--tserver_master_addrs=127.27.196.126:34625
--webserver_port=45843
--webserver_interface=127.27.196.69
--builtin_ntp_servers=127.27.196.84:38081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 21:29:05.001546 11009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:05.001927 11009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:29:05.002100 11009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:05.011415 11009 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:29:05.011575 11009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:29:05.011709 11009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.69
I20240502 21:29:05.021883 11009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:38081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.69:42049
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.27.196.69
--webserver_port=45843
--tserver_master_addrs=127.27.196.126:34625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.69
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:29:05.023660 11009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:29:05.026471 11009 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:29:05.043488 11015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:05.043805 11018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:05.044293 11016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:29:05.045027 11009 server_base.cc:1034] running on GCE node
I20240502 21:29:05.045945 11009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:29:05.047104 11009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:29:05.048417 11009 hybrid_clock.cc:648] HybridClock initialized: now 1714685345048333 us; error 74 us; skew 500 ppm
I20240502 21:29:05.048817 11009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:29:05.051968 11009 webserver.cc:460] Webserver started at http://127.27.196.69:45843/ using document root <none> and password file <none>
I20240502 21:29:05.052920 11009 fs_manager.cc:359] Metadata directory not provided
I20240502 21:29:05.053097 11009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:29:05.060089 11009 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 21:29:05.065240 11024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:29:05.067687 11009 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.005s	sys 0.001s
I20240502 21:29:05.067957 11009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "d10e9e156ba74a97a221c85b68ab8bc4"
format_stamp: "Formatted at 2024-05-02 21:28:43 on dist-test-slave-3bs1"
I20240502 21:29:05.069221 11009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:29:05.103258 11009 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:29:05.104514 11009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:29:05.104912 11009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:29:05.106201 11009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:29:05.108167 11009 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:29:05.108260 11009 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:29:05.108356 11009 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:29:05.108399 11009 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:29:05.152626 11009 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.69:42049
I20240502 21:29:05.152693 11138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.69:42049 every 8 connection(s)
I20240502 21:29:05.154951 11009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:29:05.158071 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 11009
I20240502 21:29:05.159040 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.70:44211
--local_ip_for_outbound_sockets=127.27.196.70
--tserver_master_addrs=127.27.196.126:34625
--webserver_port=43831
--webserver_interface=127.27.196.70
--builtin_ntp_servers=127.27.196.84:38081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:29:05.176801 11139 heartbeater.cc:347] Connected to a master server at 127.27.196.126:34625
I20240502 21:29:05.177347 11139 heartbeater.cc:464] Registering TS with master...
I20240502 21:29:05.178412 11139 heartbeater.cc:511] Master 127.27.196.126:34625 requested a full tablet report, sending...
I20240502 21:29:05.180394  8959 ts_manager.cc:194] Re-registered known tserver with Master: d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
I20240502 21:29:05.181871  8959 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.69:35251
W20240502 21:29:05.634651 11143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:05.635532 11143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:29:05.635948 11143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:05.649658 11143 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:29:05.649843 11143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:29:05.650053 11143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.70
I20240502 21:29:05.661381 11143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:38081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.70:44211
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.27.196.70
--webserver_port=43831
--tserver_master_addrs=127.27.196.126:34625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.70
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:29:05.662928 11143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:29:05.665081 11143 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:29:05.684016 11149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:05.684398 11152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:05.684677 11150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:29:05.686555 11143 server_base.cc:1034] running on GCE node
I20240502 21:29:05.687422 11143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:29:05.688609 11143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:29:05.689926 11143 hybrid_clock.cc:648] HybridClock initialized: now 1714685345689847 us; error 70 us; skew 500 ppm
I20240502 21:29:05.690361 11143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:29:05.693418 11143 webserver.cc:460] Webserver started at http://127.27.196.70:43831/ using document root <none> and password file <none>
I20240502 21:29:05.694429 11143 fs_manager.cc:359] Metadata directory not provided
I20240502 21:29:05.694622 11143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:29:05.703658 11143 fs_manager.cc:709] Time spent opening directory manager: real 0.006s	user 0.006s	sys 0.001s
I20240502 21:29:05.708957 11159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:29:05.711071 11143 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.004s	sys 0.001s
I20240502 21:29:05.711428 11143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6ef0beaf2f6941afb6ea9f9f274517fe"
format_stamp: "Formatted at 2024-05-02 21:28:44 on dist-test-slave-3bs1"
I20240502 21:29:05.712484 11143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:29:05.747524 11143 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:29:05.748729 11143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:29:05.749089 11143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:29:05.750524 11143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:29:05.753044 11143 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:29:05.753155 11143 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:29:05.753403 11143 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:29:05.753469 11143 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:29:05.799250 11143 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.70:44211
I20240502 21:29:05.799547 11275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.70:44211 every 8 connection(s)
I20240502 21:29:05.800992 11143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 21:29:05.807760 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 11143
I20240502 21:29:05.808615 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.71:35857
--local_ip_for_outbound_sockets=127.27.196.71
--tserver_master_addrs=127.27.196.126:34625
--webserver_port=39053
--webserver_interface=127.27.196.71
--builtin_ntp_servers=127.27.196.84:38081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:29:05.822906 11276 heartbeater.cc:347] Connected to a master server at 127.27.196.126:34625
I20240502 21:29:05.823407 11276 heartbeater.cc:464] Registering TS with master...
I20240502 21:29:05.824724 11276 heartbeater.cc:511] Master 127.27.196.126:34625 requested a full tablet report, sending...
I20240502 21:29:05.827564  8959 ts_manager.cc:194] Re-registered known tserver with Master: 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211)
I20240502 21:29:05.828927  8959 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.70:36347
I20240502 21:29:06.185657 11139 heartbeater.cc:503] Master 127.27.196.126:34625 was elected leader, sending a full tablet report...
W20240502 21:29:06.305346 11280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:06.305720 11280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:29:06.305820 11280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:06.314982 11280 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:29:06.315235 11280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:29:06.315443 11280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.71
I20240502 21:29:06.326368 11280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:38081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.71:35857
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.27.196.71
--webserver_port=39053
--tserver_master_addrs=127.27.196.126:34625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.71
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:29:06.328119 11280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:29:06.330431 11280 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:29:06.347307 11286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:06.347481 11288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:06.348057 11285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:29:06.348582 11280 server_base.cc:1034] running on GCE node
I20240502 21:29:06.349366 11280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:29:06.350714 11280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:29:06.351965 11280 hybrid_clock.cc:648] HybridClock initialized: now 1714685346351891 us; error 107 us; skew 500 ppm
I20240502 21:29:06.352375 11280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:29:06.355497 11280 webserver.cc:460] Webserver started at http://127.27.196.71:39053/ using document root <none> and password file <none>
I20240502 21:29:06.356521 11280 fs_manager.cc:359] Metadata directory not provided
I20240502 21:29:06.356698 11280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:29:06.364138 11280 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 21:29:06.367735 11294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:29:06.369570 11280 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.001s
I20240502 21:29:06.369779 11280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "4cbb5b06a6a44eb6b2145cd538b01af2"
format_stamp: "Formatted at 2024-05-02 21:28:45 on dist-test-slave-3bs1"
I20240502 21:29:06.370801 11280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:29:06.398392 11280 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:29:06.399716 11280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:29:06.400080 11280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:29:06.401693 11280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:29:06.404323 11280 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:29:06.404422 11280 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:29:06.404564 11280 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:29:06.404621 11280 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:29:06.453936 11280 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.71:35857
I20240502 21:29:06.453980 11409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.71:35857 every 8 connection(s)
I20240502 21:29:06.455644 11280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 21:29:06.465209 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 11280
I20240502 21:29:06.465997 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.72:40467
--local_ip_for_outbound_sockets=127.27.196.72
--tserver_master_addrs=127.27.196.126:34625
--webserver_port=33717
--webserver_interface=127.27.196.72
--builtin_ntp_servers=127.27.196.84:38081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:29:06.472697 11410 heartbeater.cc:347] Connected to a master server at 127.27.196.126:34625
I20240502 21:29:06.473104 11410 heartbeater.cc:464] Registering TS with master...
I20240502 21:29:06.474189 11410 heartbeater.cc:511] Master 127.27.196.126:34625 requested a full tablet report, sending...
I20240502 21:29:06.476163  8959 ts_manager.cc:194] Re-registered known tserver with Master: 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857)
I20240502 21:29:06.477479  8959 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.71:54203
I20240502 21:29:06.834368 11276 heartbeater.cc:503] Master 127.27.196.126:34625 was elected leader, sending a full tablet report...
W20240502 21:29:06.931886 11414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:06.932332 11414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:29:06.932479 11414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:06.943688 11414 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:29:06.943850 11414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:29:06.943971 11414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.72
I20240502 21:29:06.955180 11414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:38081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.72:40467
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.27.196.72
--webserver_port=33717
--tserver_master_addrs=127.27.196.126:34625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.72
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:29:06.956840 11414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:29:06.958892 11414 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:29:06.974895 11422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:06.975529 11419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:06.976341 11420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:29:06.979763 11414 server_base.cc:1034] running on GCE node
I20240502 21:29:06.981252 11414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:29:06.982973 11414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:29:06.984467 11414 hybrid_clock.cc:648] HybridClock initialized: now 1714685346984302 us; error 131 us; skew 500 ppm
I20240502 21:29:06.985067 11414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:29:06.992815 11414 webserver.cc:460] Webserver started at http://127.27.196.72:33717/ using document root <none> and password file <none>
I20240502 21:29:06.993700 11414 fs_manager.cc:359] Metadata directory not provided
I20240502 21:29:06.993990 11414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:29:07.001369 11414 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240502 21:29:07.004798 11428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:29:07.006673 11414 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
I20240502 21:29:07.006896 11414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "e0de0545fa0547d3929111e990e6d1b2"
format_stamp: "Formatted at 2024-05-02 21:28:45 on dist-test-slave-3bs1"
I20240502 21:29:07.007956 11414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:29:07.025506 11414 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:29:07.026500 11414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:29:07.026824 11414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:29:07.028275 11414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:29:07.030503 11414 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:29:07.030603 11414 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:29:07.030746 11414 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:29:07.030809 11414 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:29:07.084031 11414 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.72:40467
I20240502 21:29:07.084080 11542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.72:40467 every 8 connection(s)
I20240502 21:29:07.085930 11414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 21:29:07.089033 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 11414
I20240502 21:29:07.107379 11543 heartbeater.cc:347] Connected to a master server at 127.27.196.126:34625
I20240502 21:29:07.107887 11543 heartbeater.cc:464] Registering TS with master...
I20240502 21:29:07.109117 11543 heartbeater.cc:511] Master 127.27.196.126:34625 requested a full tablet report, sending...
I20240502 21:29:07.111289  8959 ts_manager.cc:194] Re-registered known tserver with Master: e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467)
I20240502 21:29:07.112705  8959 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.72:51347
W20240502 21:29:07.364166  9310 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.65:42177, user_credentials={real_user=slave}} blocked reactor thread for 52491us
I20240502 21:29:07.481135 11410 heartbeater.cc:503] Master 127.27.196.126:34625 was elected leader, sending a full tablet report...
I20240502 21:29:08.122903 11543 heartbeater.cc:503] Master 127.27.196.126:34625 was elected leader, sending a full tablet report...
W20240502 21:29:08.582954  9172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.65:42177, user_credentials={real_user=slave}} blocked reactor thread for 68000.2us
W20240502 21:29:08.621639  9174 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.67:43203, user_credentials={real_user=slave}} blocked reactor thread for 46867.8us
I20240502 21:29:08.874578  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:08.922291  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:08.958922  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:29:09.009321 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:09.086966  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 21:29:09.114773  9038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.68:40931, user_credentials={real_user=slave}} blocked reactor thread for 66633.7us
I20240502 21:29:09.146340 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:09.166110  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:29:09.170545 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:09.206033 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:09.210814  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:09.257835 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:09.272333 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:09.272503 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:09.334290 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:09.335067  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:09.388110  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:09.516110 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:09.528038  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:09.543313  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:09.664333 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:09.669239  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:29:09.673909 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:09.691404 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:09.720491  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:09.915995  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:09.974284  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:09.998310 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:10.041164  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:10.070102  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:29:10.076762 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:10.081934 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:10.125293  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:10.140996  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:10.163880 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:10.188016 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:10.203625  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:29:10.205183 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:10.207279 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:10.296105 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:10.345106  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:10.932143  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:10.945693 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:10.954835 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:11.023152  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:11.028085  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:11.029778 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:11.047749 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:11.088771  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:29:11.389981  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:11.486965 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:11.503867  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:29:11.521142  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:11.539947 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:11.572613 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:11.624686 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:11.648685  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:11.689568  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:11.741422 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:11.746865 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:11.746925  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:29:11.768536  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:11.908448  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:11.936347 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:11.952198 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:12.179684 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:12.196686  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:12.206625  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:29:12.242882  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:12.271471 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:12.284626 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:12.329839 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:12.351075  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 21:29:12.411139  9313 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.66:39071, user_credentials={real_user=slave}} blocked reactor thread for 49545us
I20240502 21:29:12.464639  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:12.486755 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:12.499338  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:29:12.506733 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:12.549104  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:12.553232  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:12.553560 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:12.582252 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:12.950340  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:12.993649 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:13.027446  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:13.192466  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:29:13.201119  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:13.203135 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:13.222976 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:13.231076 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:13.308494  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:13.400658  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:13.420671 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:13.422966 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:13.436540  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:29:13.478327 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:13.498100 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:13.527426  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:13.566085 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:13.567831  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:13.604195  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:29:13.652004  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:13.659865 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:13.676342 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:13.730489 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:13.749882  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 21:29:14.181365  9446 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.67:43203, user_credentials={real_user=slave}} blocked reactor thread for 49379.8us
I20240502 21:29:14.195333  9240 consensus_queue.cc:237] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:14.208781  9108 raft_consensus.cc:1261] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:14.209611  9377 raft_consensus.cc:1261] T 4d1a31513bb64578a85adf1f622e5bc0 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:14.209722 12195 consensus_queue.cc:1035] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:14.211661 12195 consensus_queue.cc:1035] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:14.268625 12195 raft_consensus.cc:2932] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:14.272492  9106 raft_consensus.cc:2932] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:14.287485  8958 catalog_manager.cc:5484] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:14.291503  9377 raft_consensus.cc:2932] T 4d1a31513bb64578a85adf1f622e5bc0 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:14.309079  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:14.374339 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:14.400225  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:14.437098 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:14.453891  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:14.447018 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:14.464048 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:14.484568  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:14.523435  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:29:14.525573 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:14.540947 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 21:29:14.568778  9173 consensus_peers.cc:485] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4d1a31513bb64578a85adf1f622e5bc0. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:14.594137  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:29:14.600437 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 21:29:14.611425  9174 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.67:43203, user_credentials={real_user=slave}} blocked reactor thread for 122581us
W20240502 21:29:14.624281  9313 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.66:39071, user_credentials={real_user=slave}} blocked reactor thread for 51914.6us
I20240502 21:29:14.631219 12391 ts_tablet_manager.cc:902] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: Initiating tablet copy from peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:29:14.668984  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:14.681012  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:14.681443 12391 tablet_copy_client.cc:308] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:39071
W20240502 21:29:14.710556  9040 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.67:43203, user_credentials={real_user=slave}} blocked reactor thread for 50237.1us
I20240502 21:29:14.739765  9263 tablet_copy_service.cc:140] P 525a093ea5bf40dc9bbf9dc39895d670: Received BeginTabletCopySession request for tablet 4d1a31513bb64578a85adf1f622e5bc0 from peer 6ef0beaf2f6941afb6ea9f9f274517fe ({username='slave'} at 127.27.196.70:52575)
I20240502 21:29:14.740315  9263 tablet_copy_service.cc:161] P 525a093ea5bf40dc9bbf9dc39895d670: Beginning new tablet copy session on tablet 4d1a31513bb64578a85adf1f622e5bc0 from peer 6ef0beaf2f6941afb6ea9f9f274517fe at {username='slave'} at 127.27.196.70:52575: session id = 6ef0beaf2f6941afb6ea9f9f274517fe-4d1a31513bb64578a85adf1f622e5bc0
I20240502 21:29:14.745890  9263 tablet_copy_source_session.cc:205] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:14.755493 12391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d1a31513bb64578a85adf1f622e5bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:14.791690 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:14.885957 12391 tablet_copy_client.cc:783] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:14.886715 12391 tablet_copy_client.cc:647] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:14.895702 12391 tablet_copy_client.cc:523] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:14.897926  9240 consensus_queue.cc:237] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:14.910607  9242 consensus_queue.cc:237] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:14.915933  9507 raft_consensus.cc:1261] T 51936fb1f678458a8795f8246590addb P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:14.915937  9106 raft_consensus.cc:1261] T 51936fb1f678458a8795f8246590addb P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:14.916972 12380 consensus_queue.cc:1035] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:14.918706 12380 consensus_queue.cc:1035] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:29:14.939556  9173 consensus_peers.cc:485] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Couldn't send request to peer d10e9e156ba74a97a221c85b68ab8bc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51936fb1f678458a8795f8246590addb. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:14.970425 12391 tablet_bootstrap.cc:492] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap starting.
I20240502 21:29:14.976115 12380 raft_consensus.cc:2932] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
W20240502 21:29:14.981659  9173 consensus_peers.cc:485] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07d693b680a54e12871121737e3da744. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:14.982515  9106 raft_consensus.cc:2932] T 51936fb1f678458a8795f8246590addb P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:14.988212  9510 raft_consensus.cc:2932] T 51936fb1f678458a8795f8246590addb P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:14.989526  8959 catalog_manager.cc:5484] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: config changed from index -1 to 4, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:14.998288  9507 raft_consensus.cc:1261] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:14.998312  9377 raft_consensus.cc:1261] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:14.999500 12381 consensus_queue.cc:1035] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:14.999929 12380 consensus_queue.cc:1035] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:15.045356 12391 log.cc:824] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: Log is configured to *not* fsync() on all Append() calls
I20240502 21:29:15.045362 12381 raft_consensus.cc:2932] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:15.059935  9378 raft_consensus.cc:2932] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:15.094645  8958 catalog_manager.cc:5484] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:15.101665  9510 raft_consensus.cc:2932] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:15.210280 12391 tablet_bootstrap.cc:492] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:15.211069 12391 tablet_bootstrap.cc:492] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap complete.
I20240502 21:29:15.211796 12391 ts_tablet_manager.cc:1367] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent bootstrapping tablet: real 0.242s	user 0.096s	sys 0.016s
I20240502 21:29:15.226981 12391 raft_consensus.cc:348] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:15.227909 12391 raft_consensus.cc:729] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef0beaf2f6941afb6ea9f9f274517fe, State: Initialized, Role: FOLLOWER
I20240502 21:29:15.228951 12391 consensus_queue.cc:260] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:15.238538 12391 ts_tablet_manager.cc:1398] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent starting tablet: real 0.027s	user 0.011s	sys 0.008s
I20240502 21:29:15.239244 11229 raft_consensus.cc:1203] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:29:15.241204  9263 tablet_copy_service.cc:342] P 525a093ea5bf40dc9bbf9dc39895d670: Request end of tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-4d1a31513bb64578a85adf1f622e5bc0 received from {username='slave'} at 127.27.196.70:52575
I20240502 21:29:15.241509  9263 tablet_copy_service.cc:433] P 525a093ea5bf40dc9bbf9dc39895d670: ending tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-4d1a31513bb64578a85adf1f622e5bc0 on tablet 4d1a31513bb64578a85adf1f622e5bc0 with peer 6ef0beaf2f6941afb6ea9f9f274517fe
I20240502 21:29:15.351965  9106 consensus_queue.cc:237] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:15.368768  9377 raft_consensus.cc:1261] T 8b4486f03fd9474aa0bb2cb4c9149437 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:15.372915 12398 consensus_queue.cc:1035] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:15.418923  9510 raft_consensus.cc:1261] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:15.423977 12398 consensus_queue.cc:1035] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:15.462277  9377 consensus_queue.cc:237] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:15.504782  9240 consensus_queue.cc:237] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
W20240502 21:29:15.510994  9173 consensus_peers.cc:485] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07d693b680a54e12871121737e3da744. This is attempt 1: this message will repeat every 5th retry.
W20240502 21:29:15.511721  9173 consensus_peers.cc:485] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Couldn't send request to peer d10e9e156ba74a97a221c85b68ab8bc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07d693b680a54e12871121737e3da744. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:15.513833  9378 raft_consensus.cc:1261] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:15.515087 12381 consensus_queue.cc:1035] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:15.524281  9507 raft_consensus.cc:1261] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:15.525338 12381 consensus_queue.cc:1035] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:15.531368 12380 raft_consensus.cc:2932] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:15.532119  9507 raft_consensus.cc:2932] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:15.541281  8959 catalog_manager.cc:5484] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: config changed from index 4 to 5, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:15.541743  9378 raft_consensus.cc:2932] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:15.554791 12513 ts_tablet_manager.cc:902] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4: Initiating tablet copy from peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
W20240502 21:29:15.570698  9447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.66:39071, user_credentials={real_user=slave}} blocked reactor thread for 122584us
I20240502 21:29:15.586504  9106 raft_consensus.cc:1261] T aeee59c5512d448d8f986f959dce1cc8 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:15.587813 12098 consensus_queue.cc:1035] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:15.589170  9510 raft_consensus.cc:1261] T aeee59c5512d448d8f986f959dce1cc8 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:15.592142 12497 raft_consensus.cc:2932] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
W20240502 21:29:15.599762  9038 consensus_peers.cc:485] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 -> Peer 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857): Couldn't send request to peer 4cbb5b06a6a44eb6b2145cd538b01af2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b4486f03fd9474aa0bb2cb4c9149437. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:15.600338 12098 consensus_queue.cc:1035] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:15.601523  9378 raft_consensus.cc:2932] T 8b4486f03fd9474aa0bb2cb4c9149437 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:15.604820  8959 catalog_manager.cc:5484] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index -1 to 4, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
W20240502 21:29:15.632005  9311 consensus_peers.cc:485] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Couldn't send request to peer d10e9e156ba74a97a221c85b68ab8bc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aeee59c5512d448d8f986f959dce1cc8. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:15.642221  9240 consensus_queue.cc:237] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:15.651473  9510 raft_consensus.cc:2932] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
W20240502 21:29:15.659391  9173 consensus_peers.cc:485] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 58c7004ceb044f5fa9dec54910046b69. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:15.661010  9378 raft_consensus.cc:1261] T 58c7004ceb044f5fa9dec54910046b69 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:15.662055 12381 consensus_queue.cc:1035] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:15.665262 12513 tablet_copy_client.cc:308] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:39071
I20240502 21:29:15.669948  9506 raft_consensus.cc:1261] T 58c7004ceb044f5fa9dec54910046b69 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:15.671916 12381 consensus_queue.cc:1035] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:15.701913 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:15.704584 12380 raft_consensus.cc:2932] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:15.709314  9508 raft_consensus.cc:2932] T 58c7004ceb044f5fa9dec54910046b69 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:15.726356  9377 raft_consensus.cc:2932] T 58c7004ceb044f5fa9dec54910046b69 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:15.730830  8959 catalog_manager.cc:5484] T 58c7004ceb044f5fa9dec54910046b69 P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:15.746150  9263 tablet_copy_service.cc:140] P 525a093ea5bf40dc9bbf9dc39895d670: Received BeginTabletCopySession request for tablet 51936fb1f678458a8795f8246590addb from peer d10e9e156ba74a97a221c85b68ab8bc4 ({username='slave'} at 127.27.196.69:46313)
I20240502 21:29:15.746475  9263 tablet_copy_service.cc:161] P 525a093ea5bf40dc9bbf9dc39895d670: Beginning new tablet copy session on tablet 51936fb1f678458a8795f8246590addb from peer d10e9e156ba74a97a221c85b68ab8bc4 at {username='slave'} at 127.27.196.69:46313: session id = d10e9e156ba74a97a221c85b68ab8bc4-51936fb1f678458a8795f8246590addb
I20240502 21:29:15.750312 12526 raft_consensus.cc:2932] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:15.751549  9263 tablet_copy_source_session.cc:205] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:15.758646 12513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51936fb1f678458a8795f8246590addb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:15.785817  9108 raft_consensus.cc:2932] T aeee59c5512d448d8f986f959dce1cc8 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:15.788730  8959 catalog_manager.cc:5484] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: config changed from index -1 to 4, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:15.794112 12513 tablet_copy_client.cc:783] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:15.794713 12513 tablet_copy_client.cc:647] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:15.799623 12513 tablet_copy_client.cc:523] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:15.815819  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:29:15.815552  9508 raft_consensus.cc:2932] T aeee59c5512d448d8f986f959dce1cc8 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:15.836081 12513 tablet_bootstrap.cc:492] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap starting.
I20240502 21:29:15.847608  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:15.867511 12552 ts_tablet_manager.cc:902] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2: Initiating tablet copy from peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:29:15.885696 12513 log.cc:824] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4: Log is configured to *not* fsync() on all Append() calls
I20240502 21:29:15.898500  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:15.975345 12552 tablet_copy_client.cc:308] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:42177
I20240502 21:29:16.011281  9128 tablet_copy_service.cc:140] P 5543ddab5b484642b139b7c848932a58: Received BeginTabletCopySession request for tablet 8b4486f03fd9474aa0bb2cb4c9149437 from peer 4cbb5b06a6a44eb6b2145cd538b01af2 ({username='slave'} at 127.27.196.71:58633)
I20240502 21:29:16.011668  9128 tablet_copy_service.cc:161] P 5543ddab5b484642b139b7c848932a58: Beginning new tablet copy session on tablet 8b4486f03fd9474aa0bb2cb4c9149437 from peer 4cbb5b06a6a44eb6b2145cd538b01af2 at {username='slave'} at 127.27.196.71:58633: session id = 4cbb5b06a6a44eb6b2145cd538b01af2-8b4486f03fd9474aa0bb2cb4c9149437
I20240502 21:29:16.022912 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:16.035601  9128 tablet_copy_source_session.cc:205] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:16.061640 12552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b4486f03fd9474aa0bb2cb4c9149437. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:16.078135 12603 ts_tablet_manager.cc:902] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe: Initiating tablet copy from peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:29:16.080515 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:16.082093 12593 ts_tablet_manager.cc:902] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4: Initiating tablet copy from peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:29:16.082340 12596 ts_tablet_manager.cc:902] T aeee59c5512d448d8f986f959dce1cc8 P d10e9e156ba74a97a221c85b68ab8bc4: Initiating tablet copy from peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:29:16.092665 12598 ts_tablet_manager.cc:902] T 58c7004ceb044f5fa9dec54910046b69 P 6ef0beaf2f6941afb6ea9f9f274517fe: Initiating tablet copy from peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:29:16.123441  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:16.139463  9510 consensus_queue.cc:237] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:16.158515 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:16.160099 12513 tablet_bootstrap.cc:492] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:16.161094 12513 tablet_bootstrap.cc:492] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap complete.
I20240502 21:29:16.161816 12513 ts_tablet_manager.cc:1367] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4: Time spent bootstrapping tablet: real 0.326s	user 0.075s	sys 0.007s
I20240502 21:29:16.181839  9378 raft_consensus.cc:1261] T a2f3edaf1c874cefb820f027f90d1185 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:16.182586  9108 raft_consensus.cc:1261] T a2f3edaf1c874cefb820f027f90d1185 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:16.183326 12510 consensus_queue.cc:1035] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:16.183944 12050 consensus_queue.cc:1035] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:16.187295 12596 tablet_copy_client.cc:308] T aeee59c5512d448d8f986f959dce1cc8 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:43203
I20240502 21:29:16.189533 12593 tablet_copy_client.cc:308] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:39071
I20240502 21:29:16.190678  9263 tablet_copy_service.cc:140] P 525a093ea5bf40dc9bbf9dc39895d670: Received BeginTabletCopySession request for tablet 07d693b680a54e12871121737e3da744 from peer d10e9e156ba74a97a221c85b68ab8bc4 ({username='slave'} at 127.27.196.69:46313)
I20240502 21:29:16.190898  9263 tablet_copy_service.cc:161] P 525a093ea5bf40dc9bbf9dc39895d670: Beginning new tablet copy session on tablet 07d693b680a54e12871121737e3da744 from peer d10e9e156ba74a97a221c85b68ab8bc4 at {username='slave'} at 127.27.196.69:46313: session id = d10e9e156ba74a97a221c85b68ab8bc4-07d693b680a54e12871121737e3da744
I20240502 21:29:16.194764  9263 tablet_copy_source_session.cc:205] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:16.200438 12593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d693b680a54e12871121737e3da744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:16.211135 12603 tablet_copy_client.cc:308] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:39071
I20240502 21:29:16.213932  9263 tablet_copy_service.cc:140] P 525a093ea5bf40dc9bbf9dc39895d670: Received BeginTabletCopySession request for tablet 07d693b680a54e12871121737e3da744 from peer 6ef0beaf2f6941afb6ea9f9f274517fe ({username='slave'} at 127.27.196.70:52575)
I20240502 21:29:16.214231  9263 tablet_copy_service.cc:161] P 525a093ea5bf40dc9bbf9dc39895d670: Beginning new tablet copy session on tablet 07d693b680a54e12871121737e3da744 from peer 6ef0beaf2f6941afb6ea9f9f274517fe at {username='slave'} at 127.27.196.70:52575: session id = 6ef0beaf2f6941afb6ea9f9f274517fe-07d693b680a54e12871121737e3da744
I20240502 21:29:16.217907  9263 tablet_copy_source_session.cc:205] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:16.223693 12513 raft_consensus.cc:348] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:16.226064 12603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d693b680a54e12871121737e3da744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:16.229625  9377 consensus_queue.cc:237] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:16.241554 12513 raft_consensus.cc:729] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d10e9e156ba74a97a221c85b68ab8bc4, State: Initialized, Role: FOLLOWER
I20240502 21:29:16.243803 12513 consensus_queue.cc:260] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:16.244452  9107 consensus_queue.cc:237] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:16.249454  9399 tablet_copy_service.cc:140] P f2fdd596a8b44bbc848fe40f9f07fbe0: Received BeginTabletCopySession request for tablet aeee59c5512d448d8f986f959dce1cc8 from peer d10e9e156ba74a97a221c85b68ab8bc4 ({username='slave'} at 127.27.196.69:60341)
I20240502 21:29:16.249846  9399 tablet_copy_service.cc:161] P f2fdd596a8b44bbc848fe40f9f07fbe0: Beginning new tablet copy session on tablet aeee59c5512d448d8f986f959dce1cc8 from peer d10e9e156ba74a97a221c85b68ab8bc4 at {username='slave'} at 127.27.196.69:60341: session id = d10e9e156ba74a97a221c85b68ab8bc4-aeee59c5512d448d8f986f959dce1cc8
I20240502 21:29:16.258654  9399 tablet_copy_source_session.cc:205] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:16.260715 12593 tablet_copy_client.cc:783] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:16.261170 12593 tablet_copy_client.cc:647] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:16.266571 12596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeee59c5512d448d8f986f959dce1cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:16.266726 12510 raft_consensus.cc:2932] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:16.273674  9106 raft_consensus.cc:2932] T a2f3edaf1c874cefb820f027f90d1185 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:16.275276 12596 tablet_copy_client.cc:783] T aeee59c5512d448d8f986f959dce1cc8 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:16.275650 12596 tablet_copy_client.cc:647] T aeee59c5512d448d8f986f959dce1cc8 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:16.276896 12593 tablet_copy_client.cc:523] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:16.280939  8963 catalog_manager.cc:5484] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: config changed from index -1 to 4, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New cstate: current_term: 1 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:16.282292  9378 raft_consensus.cc:2932] T a2f3edaf1c874cefb820f027f90d1185 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:16.288076  9241 raft_consensus.cc:1261] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:16.287536 12513 ts_tablet_manager.cc:1398] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4: Time spent starting tablet: real 0.125s	user 0.013s	sys 0.000s
W20240502 21:29:16.288604  9038 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857): Couldn't send request to peer 4cbb5b06a6a44eb6b2145cd538b01af2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cdf72d25f0f847e5998acbc53979e64d. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:16.288877 11153 consensus_queue.cc:1035] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:16.289531  9263 tablet_copy_service.cc:342] P 525a093ea5bf40dc9bbf9dc39895d670: Request end of tablet copy session d10e9e156ba74a97a221c85b68ab8bc4-51936fb1f678458a8795f8246590addb received from {username='slave'} at 127.27.196.69:46313
I20240502 21:29:16.289675  9263 tablet_copy_service.cc:433] P 525a093ea5bf40dc9bbf9dc39895d670: ending tablet copy session d10e9e156ba74a97a221c85b68ab8bc4-51936fb1f678458a8795f8246590addb on tablet 51936fb1f678458a8795f8246590addb with peer d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:29:16.293265  9510 raft_consensus.cc:1261] T cdf72d25f0f847e5998acbc53979e64d P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:16.295148 12603 tablet_copy_client.cc:783] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:16.295615 12603 tablet_copy_client.cc:647] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:16.296738 12497 consensus_queue.cc:1035] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:16.305094 12598 tablet_copy_client.cc:308] T 58c7004ceb044f5fa9dec54910046b69 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:39071
I20240502 21:29:16.306246  9263 tablet_copy_service.cc:140] P 525a093ea5bf40dc9bbf9dc39895d670: Received BeginTabletCopySession request for tablet 58c7004ceb044f5fa9dec54910046b69 from peer 6ef0beaf2f6941afb6ea9f9f274517fe ({username='slave'} at 127.27.196.70:52575)
I20240502 21:29:16.306475  9263 tablet_copy_service.cc:161] P 525a093ea5bf40dc9bbf9dc39895d670: Beginning new tablet copy session on tablet 58c7004ceb044f5fa9dec54910046b69 from peer 6ef0beaf2f6941afb6ea9f9f274517fe at {username='slave'} at 127.27.196.70:52575: session id = 6ef0beaf2f6941afb6ea9f9f274517fe-58c7004ceb044f5fa9dec54910046b69
I20240502 21:29:16.312268  9263 tablet_copy_source_session.cc:205] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:16.314380 12593 tablet_bootstrap.cc:492] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap starting.
W20240502 21:29:16.315660  9311 consensus_peers.cc:485] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Couldn't send request to peer d10e9e156ba74a97a221c85b68ab8bc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aeee59c5512d448d8f986f959dce1cc8. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:16.318845 12596 tablet_copy_client.cc:523] T aeee59c5512d448d8f986f959dce1cc8 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:16.319289 12603 tablet_copy_client.cc:523] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:16.326323 11153 raft_consensus.cc:2932] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:16.335388  9241 raft_consensus.cc:2932] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:16.336237  9107 raft_consensus.cc:1261] T aeee59c5512d448d8f986f959dce1cc8 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:16.337479  9507 raft_consensus.cc:1261] T aeee59c5512d448d8f986f959dce1cc8 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:16.337436  9510 raft_consensus.cc:2932] T cdf72d25f0f847e5998acbc53979e64d P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:16.339537  8963 catalog_manager.cc:5484] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index -1 to 4, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:16.342934 12603 tablet_bootstrap.cc:492] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap starting.
I20240502 21:29:16.343300 12598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c7004ceb044f5fa9dec54910046b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:16.349359 12526 consensus_queue.cc:1035] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:16.350095 12526 consensus_queue.cc:1035] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 21:29:16.360487  9447 consensus_peers.cc:485] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467): Couldn't send request to peer e0de0545fa0547d3929111e990e6d1b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a2f3edaf1c874cefb820f027f90d1185. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:16.361325 12098 raft_consensus.cc:2932] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:16.362174  9510 raft_consensus.cc:2932] T aeee59c5512d448d8f986f959dce1cc8 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:16.371932 12596 tablet_bootstrap.cc:492] T aeee59c5512d448d8f986f959dce1cc8 P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap starting.
I20240502 21:29:16.374081 12552 tablet_copy_client.cc:783] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:16.374413 12598 tablet_copy_client.cc:783] T 58c7004ceb044f5fa9dec54910046b69 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:16.374540 12552 tablet_copy_client.cc:647] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:16.374855 12598 tablet_copy_client.cc:647] T 58c7004ceb044f5fa9dec54910046b69 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 1 WAL segments...
W20240502 21:29:16.377305  9310 consensus_peers.cc:485] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857): Couldn't send request to peer 4cbb5b06a6a44eb6b2145cd538b01af2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aeee59c5512d448d8f986f959dce1cc8. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:16.378165 12598 tablet_copy_client.cc:523] T 58c7004ceb044f5fa9dec54910046b69 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:16.381003  9107 raft_consensus.cc:2932] T aeee59c5512d448d8f986f959dce1cc8 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:16.385118 12598 tablet_bootstrap.cc:492] T 58c7004ceb044f5fa9dec54910046b69 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap starting.
I20240502 21:29:16.388108 12552 tablet_copy_client.cc:523] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:16.390622  8963 catalog_manager.cc:5484] T aeee59c5512d448d8f986f959dce1cc8 P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index 4 to 5, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:16.412195 12552 tablet_bootstrap.cc:492] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap starting.
I20240502 21:29:16.476146  9378 consensus_queue.cc:237] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:16.486135 12552 log.cc:824] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2: Log is configured to *not* fsync() on all Append() calls
I20240502 21:29:16.500814  9241 raft_consensus.cc:1261] T 41ee481a3ece47a0905e93c66dbf949d P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:16.501565 12098 consensus_queue.cc:1035] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:29:16.502146  9310 consensus_peers.cc:485] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857): Couldn't send request to peer 4cbb5b06a6a44eb6b2145cd538b01af2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41ee481a3ece47a0905e93c66dbf949d. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:16.509586  9107 raft_consensus.cc:1261] T 41ee481a3ece47a0905e93c66dbf949d P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:16.510816 12098 consensus_queue.cc:1035] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:16.534691 12098 raft_consensus.cc:2932] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:16.537591  9241 raft_consensus.cc:2932] T 41ee481a3ece47a0905e93c66dbf949d P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:16.542726 12603 tablet_bootstrap.cc:492] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:16.546157  8959 catalog_manager.cc:5484] T 41ee481a3ece47a0905e93c66dbf949d P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: config changed from index -1 to 4, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:16.547392 12603 tablet_bootstrap.cc:492] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap complete.
I20240502 21:29:16.547803 12603 ts_tablet_manager.cc:1367] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent bootstrapping tablet: real 0.205s	user 0.046s	sys 0.008s
I20240502 21:29:16.549016 12603 raft_consensus.cc:348] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:16.549469 12603 raft_consensus.cc:729] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef0beaf2f6941afb6ea9f9f274517fe, State: Initialized, Role: FOLLOWER
I20240502 21:29:16.549880 12603 consensus_queue.cc:260] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:16.555816  9107 raft_consensus.cc:2932] T 41ee481a3ece47a0905e93c66dbf949d P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:16.570999 12598 tablet_bootstrap.cc:492] T 58c7004ceb044f5fa9dec54910046b69 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:16.572664 12598 tablet_bootstrap.cc:492] T 58c7004ceb044f5fa9dec54910046b69 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap complete.
I20240502 21:29:16.573644 12598 ts_tablet_manager.cc:1367] T 58c7004ceb044f5fa9dec54910046b69 P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent bootstrapping tablet: real 0.189s	user 0.049s	sys 0.009s
I20240502 21:29:16.576239 11093 raft_consensus.cc:1203] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:29:16.575091 12598 raft_consensus.cc:348] T 58c7004ceb044f5fa9dec54910046b69 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:16.578166 12598 raft_consensus.cc:729] T 58c7004ceb044f5fa9dec54910046b69 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef0beaf2f6941afb6ea9f9f274517fe, State: Initialized, Role: FOLLOWER
I20240502 21:29:16.578877 12598 consensus_queue.cc:260] T 58c7004ceb044f5fa9dec54910046b69 P 6ef0beaf2f6941afb6ea9f9f274517fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:16.583905 12598 ts_tablet_manager.cc:1398] T 58c7004ceb044f5fa9dec54910046b69 P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent starting tablet: real 0.010s	user 0.000s	sys 0.003s
I20240502 21:29:16.586256  9263 tablet_copy_service.cc:342] P 525a093ea5bf40dc9bbf9dc39895d670: Request end of tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-58c7004ceb044f5fa9dec54910046b69 received from {username='slave'} at 127.27.196.70:52575
I20240502 21:29:16.586448  9263 tablet_copy_service.cc:433] P 525a093ea5bf40dc9bbf9dc39895d670: ending tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-58c7004ceb044f5fa9dec54910046b69 on tablet 58c7004ceb044f5fa9dec54910046b69 with peer 6ef0beaf2f6941afb6ea9f9f274517fe
I20240502 21:29:16.597481 12603 ts_tablet_manager.cc:1398] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent starting tablet: real 0.049s	user 0.008s	sys 0.014s
I20240502 21:29:16.599021  9263 tablet_copy_service.cc:342] P 525a093ea5bf40dc9bbf9dc39895d670: Request end of tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-07d693b680a54e12871121737e3da744 received from {username='slave'} at 127.27.196.70:52575
I20240502 21:29:16.599314  9263 tablet_copy_service.cc:433] P 525a093ea5bf40dc9bbf9dc39895d670: ending tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-07d693b680a54e12871121737e3da744 on tablet 07d693b680a54e12871121737e3da744 with peer 6ef0beaf2f6941afb6ea9f9f274517fe
I20240502 21:29:16.612536 12593 tablet_bootstrap.cc:492] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:16.613509 12593 tablet_bootstrap.cc:492] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap complete.
I20240502 21:29:16.614041 12593 ts_tablet_manager.cc:1367] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4: Time spent bootstrapping tablet: real 0.300s	user 0.053s	sys 0.014s
I20240502 21:29:16.616577 12593 raft_consensus.cc:348] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:16.616957 12593 raft_consensus.cc:729] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d10e9e156ba74a97a221c85b68ab8bc4, State: Initialized, Role: FOLLOWER
I20240502 21:29:16.618925 12593 consensus_queue.cc:260] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:16.639420 12593 ts_tablet_manager.cc:1398] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4: Time spent starting tablet: real 0.025s	user 0.000s	sys 0.003s
I20240502 21:29:16.652320  9263 tablet_copy_service.cc:342] P 525a093ea5bf40dc9bbf9dc39895d670: Request end of tablet copy session d10e9e156ba74a97a221c85b68ab8bc4-07d693b680a54e12871121737e3da744 received from {username='slave'} at 127.27.196.69:46313
I20240502 21:29:16.652580  9263 tablet_copy_service.cc:433] P 525a093ea5bf40dc9bbf9dc39895d670: ending tablet copy session d10e9e156ba74a97a221c85b68ab8bc4-07d693b680a54e12871121737e3da744 on tablet 07d693b680a54e12871121737e3da744 with peer d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:29:16.672122 12552 tablet_bootstrap.cc:492] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:16.675985 12552 tablet_bootstrap.cc:492] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap complete.
I20240502 21:29:16.677233 12552 ts_tablet_manager.cc:1367] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2: Time spent bootstrapping tablet: real 0.265s	user 0.070s	sys 0.006s
I20240502 21:29:16.689291 12552 raft_consensus.cc:348] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:16.693862 12552 raft_consensus.cc:729] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbb5b06a6a44eb6b2145cd538b01af2, State: Initialized, Role: FOLLOWER
I20240502 21:29:16.695549 12552 consensus_queue.cc:260] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:16.698344 12596 tablet_bootstrap.cc:492] T aeee59c5512d448d8f986f959dce1cc8 P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 21:29:16.699117 12596 tablet_bootstrap.cc:492] T aeee59c5512d448d8f986f959dce1cc8 P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap complete.
I20240502 21:29:16.699550 12596 ts_tablet_manager.cc:1367] T aeee59c5512d448d8f986f959dce1cc8 P d10e9e156ba74a97a221c85b68ab8bc4: Time spent bootstrapping tablet: real 0.328s	user 0.066s	sys 0.012s
I20240502 21:29:16.700654 12596 raft_consensus.cc:348] T aeee59c5512d448d8f986f959dce1cc8 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:16.701695 12596 raft_consensus.cc:729] T aeee59c5512d448d8f986f959dce1cc8 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d10e9e156ba74a97a221c85b68ab8bc4, State: Initialized, Role: FOLLOWER
I20240502 21:29:16.701932 12596 consensus_queue.cc:260] T aeee59c5512d448d8f986f959dce1cc8 P d10e9e156ba74a97a221c85b68ab8bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:16.702735 12596 ts_tablet_manager.cc:1398] T aeee59c5512d448d8f986f959dce1cc8 P d10e9e156ba74a97a221c85b68ab8bc4: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 21:29:16.703938 12552 ts_tablet_manager.cc:1398] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2: Time spent starting tablet: real 0.026s	user 0.010s	sys 0.002s
I20240502 21:29:16.705817  9399 tablet_copy_service.cc:342] P f2fdd596a8b44bbc848fe40f9f07fbe0: Request end of tablet copy session d10e9e156ba74a97a221c85b68ab8bc4-aeee59c5512d448d8f986f959dce1cc8 received from {username='slave'} at 127.27.196.69:60341
I20240502 21:29:16.706063  9399 tablet_copy_service.cc:433] P f2fdd596a8b44bbc848fe40f9f07fbe0: ending tablet copy session d10e9e156ba74a97a221c85b68ab8bc4-aeee59c5512d448d8f986f959dce1cc8 on tablet aeee59c5512d448d8f986f959dce1cc8 with peer d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:29:16.707422  9128 tablet_copy_service.cc:342] P 5543ddab5b484642b139b7c848932a58: Request end of tablet copy session 4cbb5b06a6a44eb6b2145cd538b01af2-8b4486f03fd9474aa0bb2cb4c9149437 received from {username='slave'} at 127.27.196.71:58633
I20240502 21:29:16.707670  9128 tablet_copy_service.cc:433] P 5543ddab5b484642b139b7c848932a58: ending tablet copy session 4cbb5b06a6a44eb6b2145cd538b01af2-8b4486f03fd9474aa0bb2cb4c9149437 on tablet 8b4486f03fd9474aa0bb2cb4c9149437 with peer 4cbb5b06a6a44eb6b2145cd538b01af2
I20240502 21:29:16.716991  9107 consensus_queue.cc:237] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:16.739569 12552 ts_tablet_manager.cc:902] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2: Initiating tablet copy from peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:29:16.745002  9510 raft_consensus.cc:1261] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:16.745368 11365 raft_consensus.cc:1261] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:16.746019 12497 consensus_queue.cc:1035] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:16.747365 11153 consensus_queue.cc:1035] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:16.755982  9378 raft_consensus.cc:1261] T 8b4486f03fd9474aa0bb2cb4c9149437 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:16.757293 11153 consensus_queue.cc:1035] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:16.782537 12721 ts_tablet_manager.cc:902] T a2f3edaf1c874cefb820f027f90d1185 P e0de0545fa0547d3929111e990e6d1b2: Initiating tablet copy from peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
W20240502 21:29:16.789808 12720 log.cc:924] Time spent T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20240502 21:29:16.791345 12552 tablet_copy_client.cc:308] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:43203
I20240502 21:29:16.791751 11153 raft_consensus.cc:2932] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:16.806404  9510 raft_consensus.cc:2932] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:16.807241 11365 raft_consensus.cc:2932] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:16.808037  8958 catalog_manager.cc:5484] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index 4 to 5, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:16.826195  9107 consensus_queue.cc:237] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:16.832082  9378 raft_consensus.cc:2932] T 8b4486f03fd9474aa0bb2cb4c9149437 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:16.851294 12721 tablet_copy_client.cc:308] T a2f3edaf1c874cefb820f027f90d1185 P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.68:40931
I20240502 21:29:16.852723  9241 raft_consensus.cc:1261] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:16.853713  9510 raft_consensus.cc:1261] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:16.856220 12497 consensus_queue.cc:1035] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:16.860754 12497 consensus_queue.cc:1035] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:16.866302  9399 tablet_copy_service.cc:140] P f2fdd596a8b44bbc848fe40f9f07fbe0: Received BeginTabletCopySession request for tablet aeee59c5512d448d8f986f959dce1cc8 from peer 4cbb5b06a6a44eb6b2145cd538b01af2 ({username='slave'} at 127.27.196.71:42243)
I20240502 21:29:16.866572  9399 tablet_copy_service.cc:161] P f2fdd596a8b44bbc848fe40f9f07fbe0: Beginning new tablet copy session on tablet aeee59c5512d448d8f986f959dce1cc8 from peer 4cbb5b06a6a44eb6b2145cd538b01af2 at {username='slave'} at 127.27.196.71:42243: session id = 4cbb5b06a6a44eb6b2145cd538b01af2-aeee59c5512d448d8f986f959dce1cc8
I20240502 21:29:16.870194  9399 tablet_copy_source_session.cc:205] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:16.874738 12552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeee59c5512d448d8f986f959dce1cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:16.878084 12755 ts_tablet_manager.cc:902] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2: Initiating tablet copy from peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
W20240502 21:29:16.897303  9039 consensus_peers.cc:485] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 -> Peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Couldn't send request to peer d10e9e156ba74a97a221c85b68ab8bc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b4486f03fd9474aa0bb2cb4c9149437. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:16.900707 12552 tablet_copy_client.cc:783] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:16.901093 12552 tablet_copy_client.cc:647] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:16.906153 12552 tablet_copy_client.cc:523] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:16.906061 12497 raft_consensus.cc:2932] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:16.913007  9241 raft_consensus.cc:2932] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:16.915730  9507 raft_consensus.cc:2932] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:16.919782  8959 catalog_manager.cc:5484] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
W20240502 21:29:16.963537  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9240163cfac446d2a14dbd9fcbbf9ff7. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:16.964768 12755 tablet_copy_client.cc:308] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:42177
I20240502 21:29:16.965111  9533 tablet_copy_service.cc:140] P 19f162e164854f2da623a4bb8f9e6bf6: Received BeginTabletCopySession request for tablet a2f3edaf1c874cefb820f027f90d1185 from peer e0de0545fa0547d3929111e990e6d1b2 ({username='slave'} at 127.27.196.72:41441)
I20240502 21:29:16.965422  9533 tablet_copy_service.cc:161] P 19f162e164854f2da623a4bb8f9e6bf6: Beginning new tablet copy session on tablet a2f3edaf1c874cefb820f027f90d1185 from peer e0de0545fa0547d3929111e990e6d1b2 at {username='slave'} at 127.27.196.72:41441: session id = e0de0545fa0547d3929111e990e6d1b2-a2f3edaf1c874cefb820f027f90d1185
I20240502 21:29:16.965914  9128 tablet_copy_service.cc:140] P 5543ddab5b484642b139b7c848932a58: Received BeginTabletCopySession request for tablet cdf72d25f0f847e5998acbc53979e64d from peer 4cbb5b06a6a44eb6b2145cd538b01af2 ({username='slave'} at 127.27.196.71:58633)
I20240502 21:29:16.966094  9128 tablet_copy_service.cc:161] P 5543ddab5b484642b139b7c848932a58: Beginning new tablet copy session on tablet cdf72d25f0f847e5998acbc53979e64d from peer 4cbb5b06a6a44eb6b2145cd538b01af2 at {username='slave'} at 127.27.196.71:58633: session id = 4cbb5b06a6a44eb6b2145cd538b01af2-cdf72d25f0f847e5998acbc53979e64d
I20240502 21:29:16.969364  9128 tablet_copy_source_session.cc:205] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:16.971771 12755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdf72d25f0f847e5998acbc53979e64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:16.973582 12552 tablet_bootstrap.cc:492] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap starting.
I20240502 21:29:16.974982 11093 raft_consensus.cc:1203] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 21:29:16.983852  9533 tablet_copy_source_session.cc:205] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:17.007102 12755 tablet_copy_client.cc:783] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:17.007627 12755 tablet_copy_client.cc:647] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:16.999001 12721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2f3edaf1c874cefb820f027f90d1185. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:17.012303 12755 tablet_copy_client.cc:523] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:17.017074 12755 tablet_bootstrap.cc:492] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap starting.
I20240502 21:29:17.032769  9507 consensus_queue.cc:237] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:17.042475 12791 ts_tablet_manager.cc:902] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2: Initiating tablet copy from peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:29:17.051180  9378 raft_consensus.cc:1261] T 1497b781f36c43b39d22a05bb524c59d P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:29:17.051879  9447 consensus_peers.cc:485] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467): Couldn't send request to peer e0de0545fa0547d3929111e990e6d1b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1497b781f36c43b39d22a05bb524c59d. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:17.052382 12630 consensus_queue.cc:1035] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:17.055855  9241 raft_consensus.cc:1261] T 1497b781f36c43b39d22a05bb524c59d P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:17.056799 12630 consensus_queue.cc:1035] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:17.084806 12721 tablet_copy_client.cc:783] T a2f3edaf1c874cefb820f027f90d1185 P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:17.085438 12721 tablet_copy_client.cc:647] T a2f3edaf1c874cefb820f027f90d1185 P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:17.090946 12721 tablet_copy_client.cc:523] T a2f3edaf1c874cefb820f027f90d1185 P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:17.106981 12809 raft_consensus.cc:2932] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:17.109608  9241 raft_consensus.cc:2932] T 1497b781f36c43b39d22a05bb524c59d P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:17.125789  8959 catalog_manager.cc:5484] T 1497b781f36c43b39d22a05bb524c59d P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: config changed from index -1 to 4, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New cstate: current_term: 1 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:17.134914 11229 raft_consensus.cc:1203] T 58c7004ceb044f5fa9dec54910046b69 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:29:17.142287 11228 raft_consensus.cc:1203] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 21:29:17.180958 12552 tablet_bootstrap.cc:492] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:17.185151 12552 tablet_bootstrap.cc:492] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap complete.
I20240502 21:29:17.185541 12552 ts_tablet_manager.cc:1367] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2: Time spent bootstrapping tablet: real 0.212s	user 0.062s	sys 0.000s
I20240502 21:29:17.186722 12552 raft_consensus.cc:348] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:17.187018 12552 raft_consensus.cc:729] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbb5b06a6a44eb6b2145cd538b01af2, State: Initialized, Role: FOLLOWER
I20240502 21:29:17.187301 12552 consensus_queue.cc:260] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:17.187929 12721 tablet_bootstrap.cc:492] T a2f3edaf1c874cefb820f027f90d1185 P e0de0545fa0547d3929111e990e6d1b2: Bootstrap starting.
I20240502 21:29:17.191542 12552 ts_tablet_manager.cc:1398] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20240502 21:29:17.193953  9399 tablet_copy_service.cc:342] P f2fdd596a8b44bbc848fe40f9f07fbe0: Request end of tablet copy session 4cbb5b06a6a44eb6b2145cd538b01af2-aeee59c5512d448d8f986f959dce1cc8 received from {username='slave'} at 127.27.196.71:42243
I20240502 21:29:17.194154  9399 tablet_copy_service.cc:433] P f2fdd596a8b44bbc848fe40f9f07fbe0: ending tablet copy session 4cbb5b06a6a44eb6b2145cd538b01af2-aeee59c5512d448d8f986f959dce1cc8 on tablet aeee59c5512d448d8f986f959dce1cc8 with peer 4cbb5b06a6a44eb6b2145cd538b01af2
I20240502 21:29:17.207696  9378 raft_consensus.cc:2932] T 1497b781f36c43b39d22a05bb524c59d P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:17.214056 12791 tablet_copy_client.cc:308] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:43203
I20240502 21:29:17.215768  9399 tablet_copy_service.cc:140] P f2fdd596a8b44bbc848fe40f9f07fbe0: Received BeginTabletCopySession request for tablet 41ee481a3ece47a0905e93c66dbf949d from peer 4cbb5b06a6a44eb6b2145cd538b01af2 ({username='slave'} at 127.27.196.71:42243)
I20240502 21:29:17.216035  9399 tablet_copy_service.cc:161] P f2fdd596a8b44bbc848fe40f9f07fbe0: Beginning new tablet copy session on tablet 41ee481a3ece47a0905e93c66dbf949d from peer 4cbb5b06a6a44eb6b2145cd538b01af2 at {username='slave'} at 127.27.196.71:42243: session id = 4cbb5b06a6a44eb6b2145cd538b01af2-41ee481a3ece47a0905e93c66dbf949d
I20240502 21:29:17.224318  9107 consensus_queue.cc:237] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:17.226467  9399 tablet_copy_source_session.cc:205] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:17.228333 12791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ee481a3ece47a0905e93c66dbf949d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:17.233358  9241 raft_consensus.cc:1261] T cffcb4f7f7754262beeb06168097f6dc P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:17.235685 11153 consensus_queue.cc:1035] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:17.239372  9377 raft_consensus.cc:1261] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:17.241031 11153 consensus_queue.cc:1035] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:17.251263 12791 tablet_copy_client.cc:783] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:17.251782 12791 tablet_copy_client.cc:647] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:17.256206 12791 tablet_copy_client.cc:523] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:17.270920 12791 tablet_bootstrap.cc:492] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap starting.
I20240502 21:29:17.275187  9378 consensus_queue.cc:237] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:17.281822 12755 tablet_bootstrap.cc:492] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:17.282536 12755 tablet_bootstrap.cc:492] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap complete.
I20240502 21:29:17.282960 12755 ts_tablet_manager.cc:1367] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2: Time spent bootstrapping tablet: real 0.266s	user 0.067s	sys 0.002s
I20240502 21:29:17.284265 12755 raft_consensus.cc:348] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:17.284633 12755 raft_consensus.cc:729] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbb5b06a6a44eb6b2145cd538b01af2, State: Initialized, Role: FOLLOWER
I20240502 21:29:17.293201 12755 consensus_queue.cc:260] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:17.295965 12755 ts_tablet_manager.cc:1398] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.000s
I20240502 21:29:17.297205  9128 tablet_copy_service.cc:342] P 5543ddab5b484642b139b7c848932a58: Request end of tablet copy session 4cbb5b06a6a44eb6b2145cd538b01af2-cdf72d25f0f847e5998acbc53979e64d received from {username='slave'} at 127.27.196.71:58633
I20240502 21:29:17.297454  9128 tablet_copy_service.cc:433] P 5543ddab5b484642b139b7c848932a58: ending tablet copy session 4cbb5b06a6a44eb6b2145cd538b01af2-cdf72d25f0f847e5998acbc53979e64d on tablet cdf72d25f0f847e5998acbc53979e64d with peer 4cbb5b06a6a44eb6b2145cd538b01af2
I20240502 21:29:17.307416 11094 raft_consensus.cc:1203] T aeee59c5512d448d8f986f959dce1cc8 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 21:29:17.308131 11094 raft_consensus.cc:2932] T aeee59c5512d448d8f986f959dce1cc8 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:17.315495 12839 ts_tablet_manager.cc:902] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4: Initiating tablet copy from peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:29:17.316838 12828 ts_tablet_manager.cc:902] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 6ef0beaf2f6941afb6ea9f9f274517fe: Initiating tablet copy from peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:29:17.327283  9107 raft_consensus.cc:1261] T a7c6b881c9ef4cdca787f065778e526f P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:17.330850  9240 raft_consensus.cc:1261] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:17.331239 12098 consensus_queue.cc:1035] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:17.337713 12721 tablet_bootstrap.cc:700] T a2f3edaf1c874cefb820f027f90d1185 P e0de0545fa0547d3929111e990e6d1b2: Time spent opening tablet: real 0.132s	user 0.004s	sys 0.000s
I20240502 21:29:17.355576 12828 tablet_copy_client.cc:308] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:42177
I20240502 21:29:17.357720 11153 raft_consensus.cc:2932] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:17.361032  9240 raft_consensus.cc:2932] T cffcb4f7f7754262beeb06168097f6dc P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:17.392676  8959 catalog_manager.cc:5484] T cffcb4f7f7754262beeb06168097f6dc P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: config changed from index -1 to 4, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
W20240502 21:29:17.396277  9041 consensus_peers.cc:485] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 -> Peer e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467): Couldn't send request to peer e0de0545fa0547d3929111e990e6d1b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cffcb4f7f7754262beeb06168097f6dc. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:17.398924  9378 raft_consensus.cc:2932] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:17.416165 12791 tablet_bootstrap.cc:492] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:17.417563 12791 tablet_bootstrap.cc:492] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap complete.
I20240502 21:29:17.418428 12791 ts_tablet_manager.cc:1367] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2: Time spent bootstrapping tablet: real 0.148s	user 0.072s	sys 0.007s
I20240502 21:29:17.419557 12791 raft_consensus.cc:348] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:17.419879 12791 raft_consensus.cc:729] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbb5b06a6a44eb6b2145cd538b01af2, State: Initialized, Role: FOLLOWER
I20240502 21:29:17.422191 12791 consensus_queue.cc:260] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:17.422384  9128 tablet_copy_service.cc:140] P 5543ddab5b484642b139b7c848932a58: Received BeginTabletCopySession request for tablet 9240163cfac446d2a14dbd9fcbbf9ff7 from peer 6ef0beaf2f6941afb6ea9f9f274517fe ({username='slave'} at 127.27.196.70:53767)
I20240502 21:29:17.422631  9128 tablet_copy_service.cc:161] P 5543ddab5b484642b139b7c848932a58: Beginning new tablet copy session on tablet 9240163cfac446d2a14dbd9fcbbf9ff7 from peer 6ef0beaf2f6941afb6ea9f9f274517fe at {username='slave'} at 127.27.196.70:53767: session id = 6ef0beaf2f6941afb6ea9f9f274517fe-9240163cfac446d2a14dbd9fcbbf9ff7
I20240502 21:29:17.423884 12791 ts_tablet_manager.cc:1398] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 21:29:17.425714  9128 tablet_copy_source_session.cc:205] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:17.429587 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9240163cfac446d2a14dbd9fcbbf9ff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:17.438287 12828 tablet_copy_client.cc:783] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:17.438884 12828 tablet_copy_client.cc:647] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:17.440943 12828 tablet_copy_client.cc:523] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 21:29:17.443771  9310 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.65:42177, user_credentials={real_user=slave}} blocked reactor thread for 115730us
I20240502 21:29:17.444839 11365 raft_consensus.cc:1203] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 21:29:17.449096 12098 consensus_queue.cc:1035] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:17.452068 11365 raft_consensus.cc:1203] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:29:17.465664 11365 raft_consensus.cc:1203] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:29:17.465611  9242 consensus_queue.cc:237] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:17.480955  9399 tablet_copy_service.cc:342] P f2fdd596a8b44bbc848fe40f9f07fbe0: Request end of tablet copy session 4cbb5b06a6a44eb6b2145cd538b01af2-41ee481a3ece47a0905e93c66dbf949d received from {username='slave'} at 127.27.196.71:42243
I20240502 21:29:17.481206  9399 tablet_copy_service.cc:433] P f2fdd596a8b44bbc848fe40f9f07fbe0: ending tablet copy session 4cbb5b06a6a44eb6b2145cd538b01af2-41ee481a3ece47a0905e93c66dbf949d on tablet 41ee481a3ece47a0905e93c66dbf949d with peer 4cbb5b06a6a44eb6b2145cd538b01af2
I20240502 21:29:17.501186 12721 log.cc:824] T a2f3edaf1c874cefb820f027f90d1185 P e0de0545fa0547d3929111e990e6d1b2: Log is configured to *not* fsync() on all Append() calls
I20240502 21:29:17.501082 12846 raft_consensus.cc:2932] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:17.503599 12828 tablet_bootstrap.cc:492] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap starting.
I20240502 21:29:17.507556  9241 raft_consensus.cc:2932] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:17.508091  9107 raft_consensus.cc:2932] T a7c6b881c9ef4cdca787f065778e526f P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
W20240502 21:29:17.518276  9313 consensus_peers.cc:485] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467): Couldn't send request to peer e0de0545fa0547d3929111e990e6d1b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7c6b881c9ef4cdca787f065778e526f. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:17.522413  8958 catalog_manager.cc:5484] T a7c6b881c9ef4cdca787f065778e526f P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index -1 to 4, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
W20240502 21:29:17.552738  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Couldn't send request to peer d10e9e156ba74a97a221c85b68ab8bc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3caa36212ba444aa88d153ff9f90122e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:17.553078  9507 raft_consensus.cc:1261] T 3caa36212ba444aa88d153ff9f90122e P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:17.554052 12381 consensus_queue.cc:1035] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:17.565959  9378 raft_consensus.cc:1261] T 3caa36212ba444aa88d153ff9f90122e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:17.566859 12381 consensus_queue.cc:1035] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:17.574939 12839 tablet_copy_client.cc:308] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:42177
I20240502 21:29:17.589448 12381 raft_consensus.cc:2932] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:17.600353  9507 raft_consensus.cc:2932] T 3caa36212ba444aa88d153ff9f90122e P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:17.608034  9128 tablet_copy_service.cc:140] P 5543ddab5b484642b139b7c848932a58: Received BeginTabletCopySession request for tablet 8b4486f03fd9474aa0bb2cb4c9149437 from peer d10e9e156ba74a97a221c85b68ab8bc4 ({username='slave'} at 127.27.196.69:40153)
I20240502 21:29:17.608526  9128 tablet_copy_service.cc:161] P 5543ddab5b484642b139b7c848932a58: Beginning new tablet copy session on tablet 8b4486f03fd9474aa0bb2cb4c9149437 from peer d10e9e156ba74a97a221c85b68ab8bc4 at {username='slave'} at 127.27.196.69:40153: session id = d10e9e156ba74a97a221c85b68ab8bc4-8b4486f03fd9474aa0bb2cb4c9149437
I20240502 21:29:17.609546  8959 catalog_manager.cc:5484] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: config changed from index -1 to 4, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:17.612391  9128 tablet_copy_source_session.cc:205] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:17.614349 12839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b4486f03fd9474aa0bb2cb4c9149437. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:17.618113  9378 raft_consensus.cc:2932] T 3caa36212ba444aa88d153ff9f90122e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:17.625243  9107 consensus_queue.cc:237] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:17.642030 12839 tablet_copy_client.cc:783] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:17.642486 12839 tablet_copy_client.cc:647] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Starting download of 1 WAL segments...
W20240502 21:29:17.648290  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32c306d724954a16be21a34aa4cc0b5f. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:17.653373  9242 raft_consensus.cc:1261] T 32c306d724954a16be21a34aa4cc0b5f P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:17.653689  9507 raft_consensus.cc:1261] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:17.654388 11153 consensus_queue.cc:1035] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:17.655148 12398 consensus_queue.cc:1035] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:17.663784 12839 tablet_copy_client.cc:523] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:17.665789 12398 raft_consensus.cc:2932] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:17.668040  9510 raft_consensus.cc:2932] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:17.669409 12721 tablet_bootstrap.cc:492] T a2f3edaf1c874cefb820f027f90d1185 P e0de0545fa0547d3929111e990e6d1b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:17.670243 12721 tablet_bootstrap.cc:492] T a2f3edaf1c874cefb820f027f90d1185 P e0de0545fa0547d3929111e990e6d1b2: Bootstrap complete.
I20240502 21:29:17.670881 12721 ts_tablet_manager.cc:1367] T a2f3edaf1c874cefb820f027f90d1185 P e0de0545fa0547d3929111e990e6d1b2: Time spent bootstrapping tablet: real 0.484s	user 0.083s	sys 0.006s
I20240502 21:29:17.672031 12905 ts_tablet_manager.cc:902] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2: Initiating tablet copy from peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
I20240502 21:29:17.676944  9242 raft_consensus.cc:2932] T 32c306d724954a16be21a34aa4cc0b5f P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:17.679003 12721 raft_consensus.cc:348] T a2f3edaf1c874cefb820f027f90d1185 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:17.679631 12721 raft_consensus.cc:729] T a2f3edaf1c874cefb820f027f90d1185 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0de0545fa0547d3929111e990e6d1b2, State: Initialized, Role: FOLLOWER
I20240502 21:29:17.680572 12721 consensus_queue.cc:260] T a2f3edaf1c874cefb820f027f90d1185 P e0de0545fa0547d3929111e990e6d1b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:17.684026  8959 catalog_manager.cc:5484] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:17.703884 12911 ts_tablet_manager.cc:902] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2: Initiating tablet copy from peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:29:17.710994 12839 tablet_bootstrap.cc:492] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap starting.
I20240502 21:29:17.715860 12721 ts_tablet_manager.cc:1398] T a2f3edaf1c874cefb820f027f90d1185 P e0de0545fa0547d3929111e990e6d1b2: Time spent starting tablet: real 0.045s	user 0.019s	sys 0.004s
I20240502 21:29:17.717301  9533 tablet_copy_service.cc:342] P 19f162e164854f2da623a4bb8f9e6bf6: Request end of tablet copy session e0de0545fa0547d3929111e990e6d1b2-a2f3edaf1c874cefb820f027f90d1185 received from {username='slave'} at 127.27.196.72:41441
I20240502 21:29:17.717545  9533 tablet_copy_service.cc:433] P 19f162e164854f2da623a4bb8f9e6bf6: ending tablet copy session e0de0545fa0547d3929111e990e6d1b2-a2f3edaf1c874cefb820f027f90d1185 on tablet a2f3edaf1c874cefb820f027f90d1185 with peer e0de0545fa0547d3929111e990e6d1b2
I20240502 21:29:17.732517 12721 ts_tablet_manager.cc:902] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2: Initiating tablet copy from peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:29:17.734690 11495 raft_consensus.cc:1203] T a2f3edaf1c874cefb820f027f90d1185 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:29:17.777627 12905 tablet_copy_client.cc:308] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.68:40931
I20240502 21:29:17.780458  9533 tablet_copy_service.cc:140] P 19f162e164854f2da623a4bb8f9e6bf6: Received BeginTabletCopySession request for tablet 1497b781f36c43b39d22a05bb524c59d from peer e0de0545fa0547d3929111e990e6d1b2 ({username='slave'} at 127.27.196.72:41441)
I20240502 21:29:17.780689  9533 tablet_copy_service.cc:161] P 19f162e164854f2da623a4bb8f9e6bf6: Beginning new tablet copy session on tablet 1497b781f36c43b39d22a05bb524c59d from peer e0de0545fa0547d3929111e990e6d1b2 at {username='slave'} at 127.27.196.72:41441: session id = e0de0545fa0547d3929111e990e6d1b2-1497b781f36c43b39d22a05bb524c59d
I20240502 21:29:17.786593  9533 tablet_copy_source_session.cc:205] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:17.789176 12905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1497b781f36c43b39d22a05bb524c59d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:17.809149 12721 tablet_copy_client.cc:308] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:43203
I20240502 21:29:17.830978 12905 tablet_copy_client.cc:783] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:17.831619 12905 tablet_copy_client.cc:647] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:17.834362 12905 tablet_copy_client.cc:523] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:17.839438 12905 tablet_bootstrap.cc:492] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2: Bootstrap starting.
I20240502 21:29:17.840857  9399 tablet_copy_service.cc:140] P f2fdd596a8b44bbc848fe40f9f07fbe0: Received BeginTabletCopySession request for tablet a7c6b881c9ef4cdca787f065778e526f from peer e0de0545fa0547d3929111e990e6d1b2 ({username='slave'} at 127.27.196.72:46235)
I20240502 21:29:17.841102  9399 tablet_copy_service.cc:161] P f2fdd596a8b44bbc848fe40f9f07fbe0: Beginning new tablet copy session on tablet a7c6b881c9ef4cdca787f065778e526f from peer e0de0545fa0547d3929111e990e6d1b2 at {username='slave'} at 127.27.196.72:46235: session id = e0de0545fa0547d3929111e990e6d1b2-a7c6b881c9ef4cdca787f065778e526f
I20240502 21:29:17.844409  9399 tablet_copy_source_session.cc:205] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:17.847307 12721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7c6b881c9ef4cdca787f065778e526f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:17.860798 12721 tablet_copy_client.cc:783] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:17.861629 12721 tablet_copy_client.cc:647] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:17.866922 12721 tablet_copy_client.cc:523] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:17.884203 12721 tablet_bootstrap.cc:492] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2: Bootstrap starting.
I20240502 21:29:17.902901 12911 tablet_copy_client.cc:308] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:42177
I20240502 21:29:17.923218  9128 tablet_copy_service.cc:140] P 5543ddab5b484642b139b7c848932a58: Received BeginTabletCopySession request for tablet cffcb4f7f7754262beeb06168097f6dc from peer e0de0545fa0547d3929111e990e6d1b2 ({username='slave'} at 127.27.196.72:35055)
I20240502 21:29:17.923455  9128 tablet_copy_service.cc:161] P 5543ddab5b484642b139b7c848932a58: Beginning new tablet copy session on tablet cffcb4f7f7754262beeb06168097f6dc from peer e0de0545fa0547d3929111e990e6d1b2 at {username='slave'} at 127.27.196.72:35055: session id = e0de0545fa0547d3929111e990e6d1b2-cffcb4f7f7754262beeb06168097f6dc
I20240502 21:29:17.926834  9128 tablet_copy_source_session.cc:205] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:17.928900 12911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cffcb4f7f7754262beeb06168097f6dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:17.939841 12911 tablet_copy_client.cc:783] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:17.941978 12911 tablet_copy_client.cc:647] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:17.944294 12911 tablet_copy_client.cc:523] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:17.948789 12911 tablet_bootstrap.cc:492] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2: Bootstrap starting.
I20240502 21:29:18.011407  9378 consensus_queue.cc:237] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:18.016240  9242 consensus_queue.cc:237] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:18.021695 12905 tablet_bootstrap.cc:492] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:18.022265 12905 tablet_bootstrap.cc:492] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2: Bootstrap complete.
I20240502 21:29:18.022907 12905 ts_tablet_manager.cc:1367] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2: Time spent bootstrapping tablet: real 0.184s	user 0.062s	sys 0.012s
W20240502 21:29:18.023480  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Couldn't send request to peer d10e9e156ba74a97a221c85b68ab8bc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a2e06253c51405fa9b6bc9e87844760. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:18.024194  9507 raft_consensus.cc:1261] T 8a2e06253c51405fa9b6bc9e87844760 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:18.024283 12905 raft_consensus.cc:348] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:18.024540 12905 raft_consensus.cc:729] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0de0545fa0547d3929111e990e6d1b2, State: Initialized, Role: FOLLOWER
I20240502 21:29:18.024809 12905 consensus_queue.cc:260] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:18.025524 12905 ts_tablet_manager.cc:1398] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 21:29:18.026036  9240 raft_consensus.cc:1261] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:18.026542 12098 consensus_queue.cc:1035] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:18.028100 12846 consensus_queue.cc:1035] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:18.040547  9510 consensus_queue.cc:237] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:18.047173  9107 raft_consensus.cc:1261] T 3d1a5050d3f944969a560994a23700ad P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:18.048154  9533 tablet_copy_service.cc:342] P 19f162e164854f2da623a4bb8f9e6bf6: Request end of tablet copy session e0de0545fa0547d3929111e990e6d1b2-1497b781f36c43b39d22a05bb524c59d received from {username='slave'} at 127.27.196.72:41441
I20240502 21:29:18.048288  9533 tablet_copy_service.cc:433] P 19f162e164854f2da623a4bb8f9e6bf6: ending tablet copy session e0de0545fa0547d3929111e990e6d1b2-1497b781f36c43b39d22a05bb524c59d on tablet 1497b781f36c43b39d22a05bb524c59d with peer e0de0545fa0547d3929111e990e6d1b2
I20240502 21:29:18.053038 12809 consensus_queue.cc:1035] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:18.059557  9378 raft_consensus.cc:1261] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:18.061028 12380 consensus_queue.cc:1035] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:18.072420  9508 raft_consensus.cc:1261] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:18.074776 12380 consensus_queue.cc:1035] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:18.085377 12911 tablet_bootstrap.cc:492] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:18.086174 12911 tablet_bootstrap.cc:492] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2: Bootstrap complete.
I20240502 21:29:18.086542 12911 ts_tablet_manager.cc:1367] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2: Time spent bootstrapping tablet: real 0.138s	user 0.053s	sys 0.006s
I20240502 21:29:18.087915 12911 raft_consensus.cc:348] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:18.088383 12911 raft_consensus.cc:729] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0de0545fa0547d3929111e990e6d1b2, State: Initialized, Role: FOLLOWER
I20240502 21:29:18.088752 12911 consensus_queue.cc:260] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:18.089445 12911 ts_tablet_manager.cc:1398] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 21:29:18.090875  9128 tablet_copy_service.cc:342] P 5543ddab5b484642b139b7c848932a58: Request end of tablet copy session e0de0545fa0547d3929111e990e6d1b2-cffcb4f7f7754262beeb06168097f6dc received from {username='slave'} at 127.27.196.72:35055
I20240502 21:29:18.091056  9128 tablet_copy_service.cc:433] P 5543ddab5b484642b139b7c848932a58: ending tablet copy session e0de0545fa0547d3929111e990e6d1b2-cffcb4f7f7754262beeb06168097f6dc on tablet cffcb4f7f7754262beeb06168097f6dc with peer e0de0545fa0547d3929111e990e6d1b2
I20240502 21:29:18.110507 12846 raft_consensus.cc:2932] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:18.116492  9506 raft_consensus.cc:2932] T 8a2e06253c51405fa9b6bc9e87844760 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:18.131297  9242 raft_consensus.cc:2932] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:18.137286 12381 raft_consensus.cc:2932] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
W20240502 21:29:18.138159  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d1a5050d3f944969a560994a23700ad. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:18.139583  8960 catalog_manager.cc:5484] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: config changed from index -1 to 4, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
W20240502 21:29:18.144544  9172 consensus_peers.cc:485] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857): Couldn't send request to peer 4cbb5b06a6a44eb6b2145cd538b01af2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87f35aff2b104c01a9d9d48a7e9c9b23. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:18.152346 11495 raft_consensus.cc:1203] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:29:18.157699  9242 raft_consensus.cc:1261] T 3d1a5050d3f944969a560994a23700ad P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:18.158660 12809 consensus_queue.cc:1035] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:18.160742  9378 raft_consensus.cc:2932] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:18.166314  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:18.164490  9507 raft_consensus.cc:2932] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:18.180697  8960 catalog_manager.cc:5484] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: config changed from index -1 to 4, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:18.196852 12839 tablet_bootstrap.cc:492] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:18.198665 12721 tablet_bootstrap.cc:492] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:18.204247 12839 tablet_bootstrap.cc:492] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap complete.
I20240502 21:29:18.204739 12839 ts_tablet_manager.cc:1367] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4: Time spent bootstrapping tablet: real 0.494s	user 0.056s	sys 0.008s
I20240502 21:29:18.205055 13013 ts_tablet_manager.cc:902] T 32c306d724954a16be21a34aa4cc0b5f P 6ef0beaf2f6941afb6ea9f9f274517fe: Initiating tablet copy from peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:29:18.205413 12721 tablet_bootstrap.cc:492] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2: Bootstrap complete.
I20240502 21:29:18.205754 12721 ts_tablet_manager.cc:1367] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2: Time spent bootstrapping tablet: real 0.322s	user 0.070s	sys 0.001s
I20240502 21:29:18.208665 12839 raft_consensus.cc:348] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:18.209009 12839 raft_consensus.cc:729] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d10e9e156ba74a97a221c85b68ab8bc4, State: Initialized, Role: FOLLOWER
I20240502 21:29:18.210057 12721 raft_consensus.cc:348] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:18.210350 12721 raft_consensus.cc:729] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0de0545fa0547d3929111e990e6d1b2, State: Initialized, Role: FOLLOWER
I20240502 21:29:18.210709 12839 consensus_queue.cc:260] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:18.211484 12721 consensus_queue.cc:260] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:18.212998 12721 ts_tablet_manager.cc:1398] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.000s
I20240502 21:29:18.213385  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:29:18.214011  9399 tablet_copy_service.cc:342] P f2fdd596a8b44bbc848fe40f9f07fbe0: Request end of tablet copy session e0de0545fa0547d3929111e990e6d1b2-a7c6b881c9ef4cdca787f065778e526f received from {username='slave'} at 127.27.196.72:46235
I20240502 21:29:18.214217  9399 tablet_copy_service.cc:433] P f2fdd596a8b44bbc848fe40f9f07fbe0: ending tablet copy session e0de0545fa0547d3929111e990e6d1b2-a7c6b881c9ef4cdca787f065778e526f on tablet a7c6b881c9ef4cdca787f065778e526f with peer e0de0545fa0547d3929111e990e6d1b2
I20240502 21:29:18.218606 12839 ts_tablet_manager.cc:1398] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.000s
I20240502 21:29:18.220396  9128 tablet_copy_service.cc:342] P 5543ddab5b484642b139b7c848932a58: Request end of tablet copy session d10e9e156ba74a97a221c85b68ab8bc4-8b4486f03fd9474aa0bb2cb4c9149437 received from {username='slave'} at 127.27.196.69:40153
I20240502 21:29:18.220641  9128 tablet_copy_service.cc:433] P 5543ddab5b484642b139b7c848932a58: ending tablet copy session d10e9e156ba74a97a221c85b68ab8bc4-8b4486f03fd9474aa0bb2cb4c9149437 on tablet 8b4486f03fd9474aa0bb2cb4c9149437 with peer d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:29:18.234412 12839 ts_tablet_manager.cc:902] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4: Initiating tablet copy from peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:29:18.247314 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:18.270838 12828 tablet_bootstrap.cc:492] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:18.271472 12828 tablet_bootstrap.cc:492] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap complete.
I20240502 21:29:18.271809 12828 ts_tablet_manager.cc:1367] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent bootstrapping tablet: real 0.768s	user 0.057s	sys 0.004s
I20240502 21:29:18.272713 12828 raft_consensus.cc:348] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:18.273018 12828 raft_consensus.cc:729] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef0beaf2f6941afb6ea9f9f274517fe, State: Initialized, Role: FOLLOWER
I20240502 21:29:18.273315 12828 consensus_queue.cc:260] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 6ef0beaf2f6941afb6ea9f9f274517fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:18.297612 12828 ts_tablet_manager.cc:1398] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent starting tablet: real 0.026s	user 0.003s	sys 0.000s
I20240502 21:29:18.298379  9242 raft_consensus.cc:2932] T 3d1a5050d3f944969a560994a23700ad P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:18.299952  9128 tablet_copy_service.cc:342] P 5543ddab5b484642b139b7c848932a58: Request end of tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-9240163cfac446d2a14dbd9fcbbf9ff7 received from {username='slave'} at 127.27.196.70:53767
I20240502 21:29:18.300122  9128 tablet_copy_service.cc:433] P 5543ddab5b484642b139b7c848932a58: ending tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-9240163cfac446d2a14dbd9fcbbf9ff7 on tablet 9240163cfac446d2a14dbd9fcbbf9ff7 with peer 6ef0beaf2f6941afb6ea9f9f274517fe
I20240502 21:29:18.308465 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:18.310655 12839 tablet_copy_client.cc:308] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:39071
I20240502 21:29:18.312426  9263 tablet_copy_service.cc:140] P 525a093ea5bf40dc9bbf9dc39895d670: Received BeginTabletCopySession request for tablet 3caa36212ba444aa88d153ff9f90122e from peer d10e9e156ba74a97a221c85b68ab8bc4 ({username='slave'} at 127.27.196.69:46313)
I20240502 21:29:18.312145  8958 catalog_manager.cc:5484] T 3d1a5050d3f944969a560994a23700ad P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New cstate: current_term: 1 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:18.312672  9263 tablet_copy_service.cc:161] P 525a093ea5bf40dc9bbf9dc39895d670: Beginning new tablet copy session on tablet 3caa36212ba444aa88d153ff9f90122e from peer d10e9e156ba74a97a221c85b68ab8bc4 at {username='slave'} at 127.27.196.69:46313: session id = d10e9e156ba74a97a221c85b68ab8bc4-3caa36212ba444aa88d153ff9f90122e
I20240502 21:29:18.313653 12989 raft_consensus.cc:2932] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:18.316450  9263 tablet_copy_source_session.cc:205] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:18.320005 12839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3caa36212ba444aa88d153ff9f90122e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:18.321530  9107 raft_consensus.cc:2932] T 3d1a5050d3f944969a560994a23700ad P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:18.326903 12839 tablet_copy_client.cc:783] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:18.327204 12839 tablet_copy_client.cc:647] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:18.329746 12839 tablet_copy_client.cc:523] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:18.340467 12839 tablet_bootstrap.cc:492] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap starting.
I20240502 21:29:18.359416  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:18.362155 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:18.397357  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:18.462280 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:18.478479 13009 ts_tablet_manager.cc:902] T 8a2e06253c51405fa9b6bc9e87844760 P d10e9e156ba74a97a221c85b68ab8bc4: Initiating tablet copy from peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:29:18.500417  9241 consensus_queue.cc:237] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:18.529531 11495 raft_consensus.cc:1203] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:29:18.535439 13049 ts_tablet_manager.cc:902] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2: Initiating tablet copy from peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:29:18.541078 11228 raft_consensus.cc:1203] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:29:18.543517 12828 ts_tablet_manager.cc:902] T 3d1a5050d3f944969a560994a23700ad P 6ef0beaf2f6941afb6ea9f9f274517fe: Initiating tablet copy from peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
I20240502 21:29:18.548983 13013 tablet_copy_client.cc:308] T 32c306d724954a16be21a34aa4cc0b5f P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:42177
I20240502 21:29:18.550148  9128 tablet_copy_service.cc:140] P 5543ddab5b484642b139b7c848932a58: Received BeginTabletCopySession request for tablet 32c306d724954a16be21a34aa4cc0b5f from peer 6ef0beaf2f6941afb6ea9f9f274517fe ({username='slave'} at 127.27.196.70:53767)
I20240502 21:29:18.550460  9128 tablet_copy_service.cc:161] P 5543ddab5b484642b139b7c848932a58: Beginning new tablet copy session on tablet 32c306d724954a16be21a34aa4cc0b5f from peer 6ef0beaf2f6941afb6ea9f9f274517fe at {username='slave'} at 127.27.196.70:53767: session id = 6ef0beaf2f6941afb6ea9f9f274517fe-32c306d724954a16be21a34aa4cc0b5f
W20240502 21:29:18.558184  9172 consensus_peers.cc:485] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857): Couldn't send request to peer 4cbb5b06a6a44eb6b2145cd538b01af2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07d693b680a54e12871121737e3da744. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:18.559006  9128 tablet_copy_source_session.cc:205] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:18.560443 11094 raft_consensus.cc:1261] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:18.561735 13013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32c306d724954a16be21a34aa4cc0b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:18.561703 13004 consensus_queue.cc:1035] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:18.573877 11228 raft_consensus.cc:1261] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:18.574942 13004 consensus_queue.cc:1035] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:18.576351 13049 tablet_copy_client.cc:308] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:39071
I20240502 21:29:18.582566 13013 tablet_copy_client.cc:783] T 32c306d724954a16be21a34aa4cc0b5f P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:18.582974 13013 tablet_copy_client.cc:647] T 32c306d724954a16be21a34aa4cc0b5f P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:18.587875 11495 raft_consensus.cc:1203] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:29:18.589789 13013 tablet_copy_client.cc:523] T 32c306d724954a16be21a34aa4cc0b5f P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:18.594177 13013 tablet_bootstrap.cc:492] T 32c306d724954a16be21a34aa4cc0b5f P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap starting.
I20240502 21:29:18.611963  9378 raft_consensus.cc:1261] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:18.612973  9508 raft_consensus.cc:1261] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:18.612993 12381 consensus_queue.cc:1035] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:18.614159 12381 consensus_queue.cc:1035] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:18.615330 13009 tablet_copy_client.cc:308] T 8a2e06253c51405fa9b6bc9e87844760 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:43203
I20240502 21:29:18.616590  9399 tablet_copy_service.cc:140] P f2fdd596a8b44bbc848fe40f9f07fbe0: Received BeginTabletCopySession request for tablet 8a2e06253c51405fa9b6bc9e87844760 from peer d10e9e156ba74a97a221c85b68ab8bc4 ({username='slave'} at 127.27.196.69:60341)
I20240502 21:29:18.616844  9399 tablet_copy_service.cc:161] P f2fdd596a8b44bbc848fe40f9f07fbe0: Beginning new tablet copy session on tablet 8a2e06253c51405fa9b6bc9e87844760 from peer d10e9e156ba74a97a221c85b68ab8bc4 at {username='slave'} at 127.27.196.69:60341: session id = d10e9e156ba74a97a221c85b68ab8bc4-8a2e06253c51405fa9b6bc9e87844760
I20240502 21:29:18.623174  9399 tablet_copy_source_session.cc:205] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:18.642298  9263 tablet_copy_service.cc:140] P 525a093ea5bf40dc9bbf9dc39895d670: Received BeginTabletCopySession request for tablet 87f35aff2b104c01a9d9d48a7e9c9b23 from peer 4cbb5b06a6a44eb6b2145cd538b01af2 ({username='slave'} at 127.27.196.71:41931)
I20240502 21:29:18.642542  9263 tablet_copy_service.cc:161] P 525a093ea5bf40dc9bbf9dc39895d670: Beginning new tablet copy session on tablet 87f35aff2b104c01a9d9d48a7e9c9b23 from peer 4cbb5b06a6a44eb6b2145cd538b01af2 at {username='slave'} at 127.27.196.71:41931: session id = 4cbb5b06a6a44eb6b2145cd538b01af2-87f35aff2b104c01a9d9d48a7e9c9b23
I20240502 21:29:18.644560 13009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2e06253c51405fa9b6bc9e87844760. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:18.644639  9510 raft_consensus.cc:2932] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:18.646318  9263 tablet_copy_source_session.cc:205] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:18.643761 12381 raft_consensus.cc:2932] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:18.652920 13049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87f35aff2b104c01a9d9d48a7e9c9b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:18.658851  8960 catalog_manager.cc:5484] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: config changed from index 5 to 6, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:18.664182 13049 tablet_copy_client.cc:783] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:18.664685 13049 tablet_copy_client.cc:647] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:18.670107 13049 tablet_copy_client.cc:523] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:18.674474  9377 raft_consensus.cc:2932] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:18.682312 13049 tablet_bootstrap.cc:492] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap starting.
I20240502 21:29:18.684198 11094 raft_consensus.cc:2932] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:18.686982 11228 raft_consensus.cc:2932] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:18.712983 11093 raft_consensus.cc:1203] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 21:29:18.720116 12839 tablet_bootstrap.cc:492] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:18.720757 12839 tablet_bootstrap.cc:492] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap complete.
I20240502 21:29:18.721140 12839 ts_tablet_manager.cc:1367] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4: Time spent bootstrapping tablet: real 0.381s	user 0.091s	sys 0.051s
I20240502 21:29:18.722198 12839 raft_consensus.cc:348] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:18.722561 12839 raft_consensus.cc:729] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d10e9e156ba74a97a221c85b68ab8bc4, State: Initialized, Role: FOLLOWER
I20240502 21:29:18.722918 12839 consensus_queue.cc:260] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:18.724704 12839 ts_tablet_manager.cc:1398] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 21:29:18.725469  9263 tablet_copy_service.cc:342] P 525a093ea5bf40dc9bbf9dc39895d670: Request end of tablet copy session d10e9e156ba74a97a221c85b68ab8bc4-3caa36212ba444aa88d153ff9f90122e received from {username='slave'} at 127.27.196.69:46313
I20240502 21:29:18.725647  9263 tablet_copy_service.cc:433] P 525a093ea5bf40dc9bbf9dc39895d670: ending tablet copy session d10e9e156ba74a97a221c85b68ab8bc4-3caa36212ba444aa88d153ff9f90122e on tablet 3caa36212ba444aa88d153ff9f90122e with peer d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:29:18.757941 13013 tablet_bootstrap.cc:492] T 32c306d724954a16be21a34aa4cc0b5f P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:18.762197 13013 tablet_bootstrap.cc:492] T 32c306d724954a16be21a34aa4cc0b5f P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap complete.
I20240502 21:29:18.763582 13013 ts_tablet_manager.cc:1367] T 32c306d724954a16be21a34aa4cc0b5f P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent bootstrapping tablet: real 0.170s	user 0.045s	sys 0.014s
I20240502 21:29:18.765913 13009 tablet_copy_client.cc:783] T 8a2e06253c51405fa9b6bc9e87844760 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:18.766299 13009 tablet_copy_client.cc:647] T 8a2e06253c51405fa9b6bc9e87844760 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:18.767416 13013 raft_consensus.cc:348] T 32c306d724954a16be21a34aa4cc0b5f P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:18.769109 13013 raft_consensus.cc:729] T 32c306d724954a16be21a34aa4cc0b5f P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef0beaf2f6941afb6ea9f9f274517fe, State: Initialized, Role: FOLLOWER
I20240502 21:29:18.769838 13013 consensus_queue.cc:260] T 32c306d724954a16be21a34aa4cc0b5f P 6ef0beaf2f6941afb6ea9f9f274517fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:18.771786 13013 ts_tablet_manager.cc:1398] T 32c306d724954a16be21a34aa4cc0b5f P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.001s
I20240502 21:29:18.773144  9128 tablet_copy_service.cc:342] P 5543ddab5b484642b139b7c848932a58: Request end of tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-32c306d724954a16be21a34aa4cc0b5f received from {username='slave'} at 127.27.196.70:53767
I20240502 21:29:18.773303  9128 tablet_copy_service.cc:433] P 5543ddab5b484642b139b7c848932a58: ending tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-32c306d724954a16be21a34aa4cc0b5f on tablet 32c306d724954a16be21a34aa4cc0b5f with peer 6ef0beaf2f6941afb6ea9f9f274517fe
I20240502 21:29:18.774245 13049 tablet_bootstrap.cc:492] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:18.774820 13049 tablet_bootstrap.cc:492] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap complete.
I20240502 21:29:18.775220 13049 ts_tablet_manager.cc:1367] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2: Time spent bootstrapping tablet: real 0.096s	user 0.056s	sys 0.004s
I20240502 21:29:18.776357 13049 raft_consensus.cc:348] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:18.776782 13049 raft_consensus.cc:729] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbb5b06a6a44eb6b2145cd538b01af2, State: Initialized, Role: FOLLOWER
I20240502 21:29:18.777125 13049 consensus_queue.cc:260] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:18.783303 12828 tablet_copy_client.cc:308] T 3d1a5050d3f944969a560994a23700ad P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.68:40931
I20240502 21:29:18.784610 13009 tablet_copy_client.cc:523] T 8a2e06253c51405fa9b6bc9e87844760 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:18.822278  9533 tablet_copy_service.cc:140] P 19f162e164854f2da623a4bb8f9e6bf6: Received BeginTabletCopySession request for tablet 3d1a5050d3f944969a560994a23700ad from peer 6ef0beaf2f6941afb6ea9f9f274517fe ({username='slave'} at 127.27.196.70:53371)
I20240502 21:29:18.822494  9533 tablet_copy_service.cc:161] P 19f162e164854f2da623a4bb8f9e6bf6: Beginning new tablet copy session on tablet 3d1a5050d3f944969a560994a23700ad from peer 6ef0beaf2f6941afb6ea9f9f274517fe at {username='slave'} at 127.27.196.70:53371: session id = 6ef0beaf2f6941afb6ea9f9f274517fe-3d1a5050d3f944969a560994a23700ad
I20240502 21:29:18.826792  9533 tablet_copy_source_session.cc:205] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:18.828670 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1a5050d3f944969a560994a23700ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:18.829454 13049 ts_tablet_manager.cc:1398] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2: Time spent starting tablet: real 0.054s	user 0.007s	sys 0.003s
I20240502 21:29:18.830884  9263 tablet_copy_service.cc:342] P 525a093ea5bf40dc9bbf9dc39895d670: Request end of tablet copy session 4cbb5b06a6a44eb6b2145cd538b01af2-87f35aff2b104c01a9d9d48a7e9c9b23 received from {username='slave'} at 127.27.196.71:41931
I20240502 21:29:18.831069  9263 tablet_copy_service.cc:433] P 525a093ea5bf40dc9bbf9dc39895d670: ending tablet copy session 4cbb5b06a6a44eb6b2145cd538b01af2-87f35aff2b104c01a9d9d48a7e9c9b23 on tablet 87f35aff2b104c01a9d9d48a7e9c9b23 with peer 4cbb5b06a6a44eb6b2145cd538b01af2
I20240502 21:29:18.839537 12828 tablet_copy_client.cc:783] T 3d1a5050d3f944969a560994a23700ad P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:18.840173 12828 tablet_copy_client.cc:647] T 3d1a5050d3f944969a560994a23700ad P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:18.843088 12828 tablet_copy_client.cc:523] T 3d1a5050d3f944969a560994a23700ad P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:18.848955 12828 tablet_bootstrap.cc:492] T 3d1a5050d3f944969a560994a23700ad P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap starting.
I20240502 21:29:18.851912  9107 consensus_queue.cc:237] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:18.856022 13009 tablet_bootstrap.cc:492] T 8a2e06253c51405fa9b6bc9e87844760 P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap starting.
W20240502 21:29:18.866531  9039 consensus_peers.cc:485] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 -> Peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Couldn't send request to peer d10e9e156ba74a97a221c85b68ab8bc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de3287296c4547688388388dd4c7e638. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:18.868065  9508 raft_consensus.cc:1261] T de3287296c4547688388388dd4c7e638 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:18.868909 13024 consensus_queue.cc:1035] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:18.878485  9377 raft_consensus.cc:1261] T de3287296c4547688388388dd4c7e638 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:18.879313 13024 consensus_queue.cc:1035] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:18.901298 13024 raft_consensus.cc:2932] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:18.905012  9378 raft_consensus.cc:2932] T de3287296c4547688388388dd4c7e638 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:18.915855  8958 catalog_manager.cc:5484] T de3287296c4547688388388dd4c7e638 P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: config changed from index -1 to 4, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:18.962249  9508 consensus_queue.cc:237] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:18.967463 11495 raft_consensus.cc:1261] T a2f3edaf1c874cefb820f027f90d1185 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:18.971287  9377 raft_consensus.cc:1261] T a2f3edaf1c874cefb820f027f90d1185 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:18.973675 12809 consensus_queue.cc:1035] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:18.974577 12809 consensus_queue.cc:1035] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:19.007869  9508 raft_consensus.cc:2932] T de3287296c4547688388388dd4c7e638 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:19.017163  9241 consensus_queue.cc:237] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:19.021162  9107 raft_consensus.cc:1261] T a2f3edaf1c874cefb820f027f90d1185 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:19.023005 12809 raft_consensus.cc:2932] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:19.024600 13129 consensus_queue.cc:1035] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 21:29:19.025384  9172 consensus_peers.cc:485] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857): Couldn't send request to peer 4cbb5b06a6a44eb6b2145cd538b01af2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8129d13163f64368b731ba4eaeaaa81c. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:19.026340  9377 raft_consensus.cc:1261] T 8129d13163f64368b731ba4eaeaaa81c P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:19.027901 13004 consensus_queue.cc:1035] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:19.032006  9508 raft_consensus.cc:1261] T 8129d13163f64368b731ba4eaeaaa81c P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:19.033635 13004 consensus_queue.cc:1035] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:19.039598 13049 ts_tablet_manager.cc:902] T 07d693b680a54e12871121737e3da744 P 4cbb5b06a6a44eb6b2145cd538b01af2: Initiating tablet copy from peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:29:19.039042  9107 raft_consensus.cc:2932] T a2f3edaf1c874cefb820f027f90d1185 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:19.044920  9378 raft_consensus.cc:2932] T a2f3edaf1c874cefb820f027f90d1185 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:19.061772  8958 catalog_manager.cc:5484] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: config changed from index 4 to 5, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New cstate: current_term: 1 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:19.081068 11496 raft_consensus.cc:2932] T a2f3edaf1c874cefb820f027f90d1185 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:19.086144 13004 raft_consensus.cc:2932] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
W20240502 21:29:19.092620  9444 consensus_peers.cc:485] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857): Couldn't send request to peer 4cbb5b06a6a44eb6b2145cd538b01af2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a2f3edaf1c874cefb820f027f90d1185. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:19.098948  9508 raft_consensus.cc:2932] T 8129d13163f64368b731ba4eaeaaa81c P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:19.100070  8960 catalog_manager.cc:5484] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: config changed from index -1 to 4, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:19.105221 11365 raft_consensus.cc:1203] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:29:19.110421 11094 raft_consensus.cc:1203] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:29:19.098044  9377 raft_consensus.cc:2932] T 8129d13163f64368b731ba4eaeaaa81c P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:19.177323 12828 tablet_bootstrap.cc:492] T 3d1a5050d3f944969a560994a23700ad P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:19.177968 12828 tablet_bootstrap.cc:492] T 3d1a5050d3f944969a560994a23700ad P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap complete.
I20240502 21:29:19.178390 12828 ts_tablet_manager.cc:1367] T 3d1a5050d3f944969a560994a23700ad P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent bootstrapping tablet: real 0.330s	user 0.082s	sys 0.038s
I20240502 21:29:19.181521 13049 tablet_copy_client.cc:308] T 07d693b680a54e12871121737e3da744 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:39071
I20240502 21:29:19.182791  9263 tablet_copy_service.cc:140] P 525a093ea5bf40dc9bbf9dc39895d670: Received BeginTabletCopySession request for tablet 07d693b680a54e12871121737e3da744 from peer 4cbb5b06a6a44eb6b2145cd538b01af2 ({username='slave'} at 127.27.196.71:41931)
I20240502 21:29:19.183068  9263 tablet_copy_service.cc:161] P 525a093ea5bf40dc9bbf9dc39895d670: Beginning new tablet copy session on tablet 07d693b680a54e12871121737e3da744 from peer 4cbb5b06a6a44eb6b2145cd538b01af2 at {username='slave'} at 127.27.196.71:41931: session id = 4cbb5b06a6a44eb6b2145cd538b01af2-07d693b680a54e12871121737e3da744
I20240502 21:29:19.184621 12828 raft_consensus.cc:348] T 3d1a5050d3f944969a560994a23700ad P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:19.184990 12828 raft_consensus.cc:729] T 3d1a5050d3f944969a560994a23700ad P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef0beaf2f6941afb6ea9f9f274517fe, State: Initialized, Role: FOLLOWER
I20240502 21:29:19.185379 12828 consensus_queue.cc:260] T 3d1a5050d3f944969a560994a23700ad P 6ef0beaf2f6941afb6ea9f9f274517fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:19.187625  9263 tablet_copy_source_session.cc:205] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:19.187928 12828 ts_tablet_manager.cc:1398] T 3d1a5050d3f944969a560994a23700ad P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.000s
I20240502 21:29:19.189046  9533 tablet_copy_service.cc:342] P 19f162e164854f2da623a4bb8f9e6bf6: Request end of tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-3d1a5050d3f944969a560994a23700ad received from {username='slave'} at 127.27.196.70:53371
I20240502 21:29:19.189267  9533 tablet_copy_service.cc:433] P 19f162e164854f2da623a4bb8f9e6bf6: ending tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-3d1a5050d3f944969a560994a23700ad on tablet 3d1a5050d3f944969a560994a23700ad with peer 6ef0beaf2f6941afb6ea9f9f274517fe
I20240502 21:29:19.196208 13009 tablet_bootstrap.cc:492] T 8a2e06253c51405fa9b6bc9e87844760 P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:19.197059 13009 tablet_bootstrap.cc:492] T 8a2e06253c51405fa9b6bc9e87844760 P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap complete.
I20240502 21:29:19.197480 13009 ts_tablet_manager.cc:1367] T 8a2e06253c51405fa9b6bc9e87844760 P d10e9e156ba74a97a221c85b68ab8bc4: Time spent bootstrapping tablet: real 0.342s	user 0.068s	sys 0.009s
I20240502 21:29:19.198704 13049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d693b680a54e12871121737e3da744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:19.198591 13009 raft_consensus.cc:348] T 8a2e06253c51405fa9b6bc9e87844760 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:19.199020 13009 raft_consensus.cc:729] T 8a2e06253c51405fa9b6bc9e87844760 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d10e9e156ba74a97a221c85b68ab8bc4, State: Initialized, Role: FOLLOWER
I20240502 21:29:19.199481 13009 consensus_queue.cc:260] T 8a2e06253c51405fa9b6bc9e87844760 P d10e9e156ba74a97a221c85b68ab8bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:19.200321 13009 ts_tablet_manager.cc:1398] T 8a2e06253c51405fa9b6bc9e87844760 P d10e9e156ba74a97a221c85b68ab8bc4: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 21:29:19.201889  9399 tablet_copy_service.cc:342] P f2fdd596a8b44bbc848fe40f9f07fbe0: Request end of tablet copy session d10e9e156ba74a97a221c85b68ab8bc4-8a2e06253c51405fa9b6bc9e87844760 received from {username='slave'} at 127.27.196.69:60341
I20240502 21:29:19.202104  9399 tablet_copy_service.cc:433] P f2fdd596a8b44bbc848fe40f9f07fbe0: ending tablet copy session d10e9e156ba74a97a221c85b68ab8bc4-8a2e06253c51405fa9b6bc9e87844760 on tablet 8a2e06253c51405fa9b6bc9e87844760 with peer d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:29:19.206676  9377 consensus_queue.cc:237] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:19.207834 11228 raft_consensus.cc:1203] T 32c306d724954a16be21a34aa4cc0b5f P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:29:19.212945  9107 raft_consensus.cc:1261] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:19.213742 12984 consensus_queue.cc:1035] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:29:19.215170  9313 consensus_peers.cc:485] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467): Couldn't send request to peer e0de0545fa0547d3929111e990e6d1b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9ae8fac693d4d68b6741f2509e1265e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:19.220628  9242 raft_consensus.cc:1261] T f9ae8fac693d4d68b6741f2509e1265e P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:19.221590 12984 consensus_queue.cc:1035] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:19.231947 13049 tablet_copy_client.cc:783] T 07d693b680a54e12871121737e3da744 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:19.232241 13049 tablet_copy_client.cc:647] T 07d693b680a54e12871121737e3da744 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:19.240012 12984 raft_consensus.cc:2932] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:19.245832  9242 raft_consensus.cc:2932] T f9ae8fac693d4d68b6741f2509e1265e P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:19.249022 13049 tablet_copy_client.cc:523] T 07d693b680a54e12871121737e3da744 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:19.273638 13049 tablet_bootstrap.cc:492] T 07d693b680a54e12871121737e3da744 P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap starting.
I20240502 21:29:19.276495  9107 raft_consensus.cc:2932] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:19.275707  8958 catalog_manager.cc:5484] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: config changed from index -1 to 4, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:19.323053 13009 ts_tablet_manager.cc:902] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4: Initiating tablet copy from peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:29:19.369197 13009 tablet_copy_client.cc:308] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:42177
I20240502 21:29:19.370091  9128 tablet_copy_service.cc:140] P 5543ddab5b484642b139b7c848932a58: Received BeginTabletCopySession request for tablet de3287296c4547688388388dd4c7e638 from peer d10e9e156ba74a97a221c85b68ab8bc4 ({username='slave'} at 127.27.196.69:40153)
I20240502 21:29:19.370297  9128 tablet_copy_service.cc:161] P 5543ddab5b484642b139b7c848932a58: Beginning new tablet copy session on tablet de3287296c4547688388388dd4c7e638 from peer d10e9e156ba74a97a221c85b68ab8bc4 at {username='slave'} at 127.27.196.69:40153: session id = d10e9e156ba74a97a221c85b68ab8bc4-de3287296c4547688388388dd4c7e638
I20240502 21:29:19.373808  9128 tablet_copy_source_session.cc:205] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:19.376150  9508 consensus_queue.cc:237] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:19.378666 13009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de3287296c4547688388388dd4c7e638. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 21:29:19.384099  9445 consensus_peers.cc:485] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f82c35077224414cb284472d49d7de4f. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:19.385962  9242 raft_consensus.cc:1261] T f82c35077224414cb284472d49d7de4f P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:19.386811 13128 consensus_queue.cc:1035] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:19.387933  9377 consensus_queue.cc:237] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:19.389941 11228 raft_consensus.cc:1203] T 3d1a5050d3f944969a560994a23700ad P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:29:19.394821  9378 raft_consensus.cc:1261] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:19.394878 11094 raft_consensus.cc:1203] T 8a2e06253c51405fa9b6bc9e87844760 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:29:19.400861 13128 consensus_queue.cc:1035] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:29:19.411895  9311 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Couldn't send request to peer d10e9e156ba74a97a221c85b68ab8bc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: be34ec89876d42b69efe1878c0d9bc19. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:19.412065  9107 raft_consensus.cc:1261] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:19.412951 13174 consensus_queue.cc:1035] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:19.437472  9508 raft_consensus.cc:1261] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:19.439147 13174 consensus_queue.cc:1035] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:19.461972 13009 tablet_copy_client.cc:783] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:19.462298 13009 tablet_copy_client.cc:647] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:19.463368 13140 raft_consensus.cc:2932] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:19.464908  9242 raft_consensus.cc:2932] T f82c35077224414cb284472d49d7de4f P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:19.467299 13009 tablet_copy_client.cc:523] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:19.476540 13174 raft_consensus.cc:2932] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:19.480636  8958 catalog_manager.cc:5484] T f82c35077224414cb284472d49d7de4f P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New cstate: current_term: 1 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:19.481595 13009 tablet_bootstrap.cc:492] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap starting.
I20240502 21:29:19.486111  8959 catalog_manager.cc:5484] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: config changed from index -1 to 4, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:19.490938  9107 raft_consensus.cc:2932] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:19.494916 13235 ts_tablet_manager.cc:902] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2: Initiating tablet copy from peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:29:19.499600  9378 raft_consensus.cc:2932] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:19.514624  9508 raft_consensus.cc:2932] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:19.578830 13235 tablet_copy_client.cc:308] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:39071
I20240502 21:29:19.580197  9263 tablet_copy_service.cc:140] P 525a093ea5bf40dc9bbf9dc39895d670: Received BeginTabletCopySession request for tablet 8129d13163f64368b731ba4eaeaaa81c from peer 4cbb5b06a6a44eb6b2145cd538b01af2 ({username='slave'} at 127.27.196.71:41931)
I20240502 21:29:19.580462  9263 tablet_copy_service.cc:161] P 525a093ea5bf40dc9bbf9dc39895d670: Beginning new tablet copy session on tablet 8129d13163f64368b731ba4eaeaaa81c from peer 4cbb5b06a6a44eb6b2145cd538b01af2 at {username='slave'} at 127.27.196.71:41931: session id = 4cbb5b06a6a44eb6b2145cd538b01af2-8129d13163f64368b731ba4eaeaaa81c
I20240502 21:29:19.581365 13259 ts_tablet_manager.cc:902] T a2f3edaf1c874cefb820f027f90d1185 P 4cbb5b06a6a44eb6b2145cd538b01af2: Initiating tablet copy from peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
I20240502 21:29:19.597265  9263 tablet_copy_source_session.cc:205] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:19.599315 13235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8129d13163f64368b731ba4eaeaaa81c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:19.607004 13235 tablet_copy_client.cc:783] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:19.607350 13235 tablet_copy_client.cc:647] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:19.609428 13235 tablet_copy_client.cc:523] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:19.613675 13235 tablet_bootstrap.cc:492] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap starting.
I20240502 21:29:19.632659 13259 tablet_copy_client.cc:308] T a2f3edaf1c874cefb820f027f90d1185 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.68:40931
I20240502 21:29:19.640548 13049 tablet_bootstrap.cc:492] T 07d693b680a54e12871121737e3da744 P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:19.641134 13049 tablet_bootstrap.cc:492] T 07d693b680a54e12871121737e3da744 P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap complete.
I20240502 21:29:19.641523 13049 ts_tablet_manager.cc:1367] T 07d693b680a54e12871121737e3da744 P 4cbb5b06a6a44eb6b2145cd538b01af2: Time spent bootstrapping tablet: real 0.368s	user 0.065s	sys 0.000s
I20240502 21:29:19.642637 13049 raft_consensus.cc:348] T 07d693b680a54e12871121737e3da744 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:19.642956 13049 raft_consensus.cc:729] T 07d693b680a54e12871121737e3da744 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbb5b06a6a44eb6b2145cd538b01af2, State: Initialized, Role: FOLLOWER
I20240502 21:29:19.657347 13049 consensus_queue.cc:260] T 07d693b680a54e12871121737e3da744 P 4cbb5b06a6a44eb6b2145cd538b01af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:19.661114  9533 tablet_copy_service.cc:140] P 19f162e164854f2da623a4bb8f9e6bf6: Received BeginTabletCopySession request for tablet a2f3edaf1c874cefb820f027f90d1185 from peer 4cbb5b06a6a44eb6b2145cd538b01af2 ({username='slave'} at 127.27.196.71:34057)
I20240502 21:29:19.661381  9533 tablet_copy_service.cc:161] P 19f162e164854f2da623a4bb8f9e6bf6: Beginning new tablet copy session on tablet a2f3edaf1c874cefb820f027f90d1185 from peer 4cbb5b06a6a44eb6b2145cd538b01af2 at {username='slave'} at 127.27.196.71:34057: session id = 4cbb5b06a6a44eb6b2145cd538b01af2-a2f3edaf1c874cefb820f027f90d1185
I20240502 21:29:19.662148 13049 ts_tablet_manager.cc:1398] T 07d693b680a54e12871121737e3da744 P 4cbb5b06a6a44eb6b2145cd538b01af2: Time spent starting tablet: real 0.021s	user 0.002s	sys 0.003s
I20240502 21:29:19.664083  9533 tablet_copy_source_session.cc:205] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:19.665234  9263 tablet_copy_service.cc:342] P 525a093ea5bf40dc9bbf9dc39895d670: Request end of tablet copy session 4cbb5b06a6a44eb6b2145cd538b01af2-07d693b680a54e12871121737e3da744 received from {username='slave'} at 127.27.196.71:41931
I20240502 21:29:19.665431  9263 tablet_copy_service.cc:433] P 525a093ea5bf40dc9bbf9dc39895d670: ending tablet copy session 4cbb5b06a6a44eb6b2145cd538b01af2-07d693b680a54e12871121737e3da744 on tablet 07d693b680a54e12871121737e3da744 with peer 4cbb5b06a6a44eb6b2145cd538b01af2
I20240502 21:29:19.667994 13259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2f3edaf1c874cefb820f027f90d1185. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:19.678061 13259 tablet_copy_client.cc:783] T a2f3edaf1c874cefb820f027f90d1185 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:19.679092 13259 tablet_copy_client.cc:647] T a2f3edaf1c874cefb820f027f90d1185 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:19.681494 13259 tablet_copy_client.cc:523] T a2f3edaf1c874cefb820f027f90d1185 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:19.689316 13259 tablet_bootstrap.cc:492] T a2f3edaf1c874cefb820f027f90d1185 P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap starting.
I20240502 21:29:19.706463 13235 tablet_bootstrap.cc:492] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:19.707091 13235 tablet_bootstrap.cc:492] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap complete.
I20240502 21:29:19.707453 13235 ts_tablet_manager.cc:1367] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2: Time spent bootstrapping tablet: real 0.094s	user 0.052s	sys 0.004s
I20240502 21:29:19.708662 13235 raft_consensus.cc:348] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:19.709028 13235 raft_consensus.cc:729] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbb5b06a6a44eb6b2145cd538b01af2, State: Initialized, Role: FOLLOWER
I20240502 21:29:19.709282 13235 consensus_queue.cc:260] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:19.709838 13235 ts_tablet_manager.cc:1398] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 21:29:19.712191  9263 tablet_copy_service.cc:342] P 525a093ea5bf40dc9bbf9dc39895d670: Request end of tablet copy session 4cbb5b06a6a44eb6b2145cd538b01af2-8129d13163f64368b731ba4eaeaaa81c received from {username='slave'} at 127.27.196.71:41931
I20240502 21:29:19.712373  9263 tablet_copy_service.cc:433] P 525a093ea5bf40dc9bbf9dc39895d670: ending tablet copy session 4cbb5b06a6a44eb6b2145cd538b01af2-8129d13163f64368b731ba4eaeaaa81c on tablet 8129d13163f64368b731ba4eaeaaa81c with peer 4cbb5b06a6a44eb6b2145cd538b01af2
I20240502 21:29:19.725890 13009 tablet_bootstrap.cc:492] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:19.726397 13009 tablet_bootstrap.cc:492] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap complete.
I20240502 21:29:19.726692 13009 ts_tablet_manager.cc:1367] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4: Time spent bootstrapping tablet: real 0.245s	user 0.060s	sys 0.000s
I20240502 21:29:19.747608 13009 raft_consensus.cc:348] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:19.747948 13009 raft_consensus.cc:729] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d10e9e156ba74a97a221c85b68ab8bc4, State: Initialized, Role: FOLLOWER
I20240502 21:29:19.748358 13009 consensus_queue.cc:260] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:19.751870 13009 ts_tablet_manager.cc:1398] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4: Time spent starting tablet: real 0.025s	user 0.005s	sys 0.000s
I20240502 21:29:19.752952  9128 tablet_copy_service.cc:342] P 5543ddab5b484642b139b7c848932a58: Request end of tablet copy session d10e9e156ba74a97a221c85b68ab8bc4-de3287296c4547688388388dd4c7e638 received from {username='slave'} at 127.27.196.69:40153
I20240502 21:29:19.753115  9128 tablet_copy_service.cc:433] P 5543ddab5b484642b139b7c848932a58: ending tablet copy session d10e9e156ba74a97a221c85b68ab8bc4-de3287296c4547688388388dd4c7e638 on tablet de3287296c4547688388388dd4c7e638 with peer d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:29:19.791069  9107 consensus_queue.cc:237] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:19.796846 11094 raft_consensus.cc:1203] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:29:19.809522 13259 tablet_bootstrap.cc:492] T a2f3edaf1c874cefb820f027f90d1185 P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:19.810067 13259 tablet_bootstrap.cc:492] T a2f3edaf1c874cefb820f027f90d1185 P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap complete.
I20240502 21:29:19.810377 13259 ts_tablet_manager.cc:1367] T a2f3edaf1c874cefb820f027f90d1185 P 4cbb5b06a6a44eb6b2145cd538b01af2: Time spent bootstrapping tablet: real 0.121s	user 0.055s	sys 0.005s
I20240502 21:29:19.811219 13259 raft_consensus.cc:348] T a2f3edaf1c874cefb820f027f90d1185 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:19.811544 13259 raft_consensus.cc:729] T a2f3edaf1c874cefb820f027f90d1185 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbb5b06a6a44eb6b2145cd538b01af2, State: Initialized, Role: FOLLOWER
I20240502 21:29:19.811828 13259 consensus_queue.cc:260] T a2f3edaf1c874cefb820f027f90d1185 P 4cbb5b06a6a44eb6b2145cd538b01af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:19.812422 13259 ts_tablet_manager.cc:1398] T a2f3edaf1c874cefb820f027f90d1185 P 4cbb5b06a6a44eb6b2145cd538b01af2: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 21:29:19.815894  9533 tablet_copy_service.cc:342] P 19f162e164854f2da623a4bb8f9e6bf6: Request end of tablet copy session 4cbb5b06a6a44eb6b2145cd538b01af2-a2f3edaf1c874cefb820f027f90d1185 received from {username='slave'} at 127.27.196.71:34057
I20240502 21:29:19.816102  9533 tablet_copy_service.cc:433] P 19f162e164854f2da623a4bb8f9e6bf6: ending tablet copy session 4cbb5b06a6a44eb6b2145cd538b01af2-a2f3edaf1c874cefb820f027f90d1185 on tablet a2f3edaf1c874cefb820f027f90d1185 with peer 4cbb5b06a6a44eb6b2145cd538b01af2
I20240502 21:29:19.832221 13299 ts_tablet_manager.cc:902] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe: Initiating tablet copy from peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
I20240502 21:29:19.843586 13299 tablet_copy_client.cc:308] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.68:40931
I20240502 21:29:19.844008 13298 ts_tablet_manager.cc:902] T f9ae8fac693d4d68b6741f2509e1265e P e0de0545fa0547d3929111e990e6d1b2: Initiating tablet copy from peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:29:19.846028  9533 tablet_copy_service.cc:140] P 19f162e164854f2da623a4bb8f9e6bf6: Received BeginTabletCopySession request for tablet f82c35077224414cb284472d49d7de4f from peer 6ef0beaf2f6941afb6ea9f9f274517fe ({username='slave'} at 127.27.196.70:53371)
I20240502 21:29:19.846236  9533 tablet_copy_service.cc:161] P 19f162e164854f2da623a4bb8f9e6bf6: Beginning new tablet copy session on tablet f82c35077224414cb284472d49d7de4f from peer 6ef0beaf2f6941afb6ea9f9f274517fe at {username='slave'} at 127.27.196.70:53371: session id = 6ef0beaf2f6941afb6ea9f9f274517fe-f82c35077224414cb284472d49d7de4f
W20240502 21:29:19.846292  9038 consensus_peers.cc:485] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 -> Peer 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857): Couldn't send request to peer 4cbb5b06a6a44eb6b2145cd538b01af2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f0259f01acb249dd99c662151dc21916. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:19.847420  9508 raft_consensus.cc:1261] T f0259f01acb249dd99c662151dc21916 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:19.848109 13024 consensus_queue.cc:1035] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:19.853392  9533 tablet_copy_source_session.cc:205] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:19.855697 13299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f82c35077224414cb284472d49d7de4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:19.861068  9378 raft_consensus.cc:1261] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:19.861949 13024 consensus_queue.cc:1035] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:19.868820 11365 raft_consensus.cc:1203] T 07d693b680a54e12871121737e3da744 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6]   Dedup: 1.6->[]
I20240502 21:29:19.869799 13299 tablet_copy_client.cc:783] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:19.870922 13299 tablet_copy_client.cc:647] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:19.876583 13009 ts_tablet_manager.cc:902] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4: Initiating tablet copy from peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:29:19.896225 13299 tablet_copy_client.cc:523] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:19.904285 13118 raft_consensus.cc:2932] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:19.905211  9508 raft_consensus.cc:2932] T f0259f01acb249dd99c662151dc21916 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:19.914165 13009 tablet_copy_client.cc:308] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:43203
I20240502 21:29:19.915429  9399 tablet_copy_service.cc:140] P f2fdd596a8b44bbc848fe40f9f07fbe0: Received BeginTabletCopySession request for tablet be34ec89876d42b69efe1878c0d9bc19 from peer d10e9e156ba74a97a221c85b68ab8bc4 ({username='slave'} at 127.27.196.69:60341)
I20240502 21:29:19.915633  9399 tablet_copy_service.cc:161] P f2fdd596a8b44bbc848fe40f9f07fbe0: Beginning new tablet copy session on tablet be34ec89876d42b69efe1878c0d9bc19 from peer d10e9e156ba74a97a221c85b68ab8bc4 at {username='slave'} at 127.27.196.69:60341: session id = d10e9e156ba74a97a221c85b68ab8bc4-be34ec89876d42b69efe1878c0d9bc19
I20240502 21:29:19.927733 13299 tablet_bootstrap.cc:492] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap starting.
I20240502 21:29:19.931092  8960 catalog_manager.cc:5484] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index -1 to 4, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:19.940057  9378 raft_consensus.cc:2932] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:19.944096  9399 tablet_copy_source_session.cc:205] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:19.948757 13009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be34ec89876d42b69efe1878c0d9bc19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:19.957696 11365 raft_consensus.cc:1203] T a2f3edaf1c874cefb820f027f90d1185 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 21:29:19.989518 13009 tablet_copy_client.cc:783] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:19.990772 13009 tablet_copy_client.cc:647] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:19.993134 13009 tablet_copy_client.cc:523] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:20.001137 13009 tablet_bootstrap.cc:492] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap starting.
I20240502 21:29:20.011750  9378 consensus_queue.cc:237] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:20.006712 13298 tablet_copy_client.cc:308] T f9ae8fac693d4d68b6741f2509e1265e P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:43203
I20240502 21:29:20.028406 11365 raft_consensus.cc:1203] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:29:20.028724  9399 tablet_copy_service.cc:140] P f2fdd596a8b44bbc848fe40f9f07fbe0: Received BeginTabletCopySession request for tablet f9ae8fac693d4d68b6741f2509e1265e from peer e0de0545fa0547d3929111e990e6d1b2 ({username='slave'} at 127.27.196.72:46235)
I20240502 21:29:20.028926  9399 tablet_copy_service.cc:161] P f2fdd596a8b44bbc848fe40f9f07fbe0: Beginning new tablet copy session on tablet f9ae8fac693d4d68b6741f2509e1265e from peer e0de0545fa0547d3929111e990e6d1b2 at {username='slave'} at 127.27.196.72:46235: session id = e0de0545fa0547d3929111e990e6d1b2-f9ae8fac693d4d68b6741f2509e1265e
I20240502 21:29:20.032073  9399 tablet_copy_source_session.cc:205] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:20.034685 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9ae8fac693d4d68b6741f2509e1265e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 21:29:20.039781  9313 consensus_peers.cc:485] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467): Couldn't send request to peer e0de0545fa0547d3929111e990e6d1b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41ee481a3ece47a0905e93c66dbf949d. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:20.040236 11365 raft_consensus.cc:1261] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:20.040750  9107 raft_consensus.cc:1261] T 41ee481a3ece47a0905e93c66dbf949d P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:20.041689 13173 consensus_queue.cc:1035] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:20.042387 13174 consensus_queue.cc:1035] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:20.042995  9242 raft_consensus.cc:1261] T 41ee481a3ece47a0905e93c66dbf949d P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:20.044763 13174 consensus_queue.cc:1035] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:20.089563 13174 raft_consensus.cc:2932] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:20.100900 11364 raft_consensus.cc:2932] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:20.101418  8959 catalog_manager.cc:5484] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: config changed from index 4 to 5, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:20.108335  9107 raft_consensus.cc:2932] T 41ee481a3ece47a0905e93c66dbf949d P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:20.118153  9242 raft_consensus.cc:2932] T 41ee481a3ece47a0905e93c66dbf949d P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:20.185194 13298 tablet_copy_client.cc:783] T f9ae8fac693d4d68b6741f2509e1265e P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:20.185626 13298 tablet_copy_client.cc:647] T f9ae8fac693d4d68b6741f2509e1265e P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:20.193612 13298 tablet_copy_client.cc:523] T f9ae8fac693d4d68b6741f2509e1265e P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:20.236794 13299 tablet_bootstrap.cc:492] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:20.237470 13299 tablet_bootstrap.cc:492] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap complete.
I20240502 21:29:20.237846 13299 ts_tablet_manager.cc:1367] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent bootstrapping tablet: real 0.310s	user 0.054s	sys 0.012s
I20240502 21:29:20.237870 13298 tablet_bootstrap.cc:492] T f9ae8fac693d4d68b6741f2509e1265e P e0de0545fa0547d3929111e990e6d1b2: Bootstrap starting.
I20240502 21:29:20.238852 13299 raft_consensus.cc:348] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:20.239311 13299 raft_consensus.cc:729] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef0beaf2f6941afb6ea9f9f274517fe, State: Initialized, Role: FOLLOWER
I20240502 21:29:20.239677 13299 consensus_queue.cc:260] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:20.253252 13299 ts_tablet_manager.cc:1398] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent starting tablet: real 0.015s	user 0.003s	sys 0.007s
I20240502 21:29:20.255049  9533 tablet_copy_service.cc:342] P 19f162e164854f2da623a4bb8f9e6bf6: Request end of tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-f82c35077224414cb284472d49d7de4f received from {username='slave'} at 127.27.196.70:53371
I20240502 21:29:20.255304  9533 tablet_copy_service.cc:433] P 19f162e164854f2da623a4bb8f9e6bf6: ending tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-f82c35077224414cb284472d49d7de4f on tablet f82c35077224414cb284472d49d7de4f with peer 6ef0beaf2f6941afb6ea9f9f274517fe
I20240502 21:29:20.337968  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:20.360285 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:20.362852  9508 consensus_queue.cc:237] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:20.366906 11228 raft_consensus.cc:1261] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 21:29:20.368777  9447 consensus_peers.cc:485] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467): Couldn't send request to peer e0de0545fa0547d3929111e990e6d1b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f82c35077224414cb284472d49d7de4f. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:20.372084 12809 consensus_queue.cc:1035] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:20.376122  9378 raft_consensus.cc:1261] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:20.377902 13140 consensus_queue.cc:1035] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:20.389104  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:20.409549  9242 raft_consensus.cc:1261] T f82c35077224414cb284472d49d7de4f P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:20.410802 13420 consensus_queue.cc:1035] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:20.416441 12809 raft_consensus.cc:2932] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:20.422799 11229 raft_consensus.cc:2932] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:20.425128 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:20.430661 13431 ts_tablet_manager.cc:902] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2: Initiating tablet copy from peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:29:20.432664  9242 raft_consensus.cc:2932] T f82c35077224414cb284472d49d7de4f P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:20.464012  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:29:20.482546 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:20.482918  8960 catalog_manager.cc:5484] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe reported cstate change: config changed from index 4 to 5, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New cstate: current_term: 1 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:20.490566  9378 raft_consensus.cc:2932] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:20.523881  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:20.565594 13431 tablet_copy_client.cc:308] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:42177
I20240502 21:29:20.566669  9128 tablet_copy_service.cc:140] P 5543ddab5b484642b139b7c848932a58: Received BeginTabletCopySession request for tablet f0259f01acb249dd99c662151dc21916 from peer 4cbb5b06a6a44eb6b2145cd538b01af2 ({username='slave'} at 127.27.196.71:58633)
I20240502 21:29:20.566897  9128 tablet_copy_service.cc:161] P 5543ddab5b484642b139b7c848932a58: Beginning new tablet copy session on tablet f0259f01acb249dd99c662151dc21916 from peer 4cbb5b06a6a44eb6b2145cd538b01af2 at {username='slave'} at 127.27.196.71:58633: session id = 4cbb5b06a6a44eb6b2145cd538b01af2-f0259f01acb249dd99c662151dc21916
I20240502 21:29:20.570076  9128 tablet_copy_source_session.cc:205] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:20.576481 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:20.604763 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:20.615052 13431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0259f01acb249dd99c662151dc21916. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:20.637032  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:20.652204 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:20.659503 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:20.670411  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:20.683782 13454 ts_tablet_manager.cc:902] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2: Initiating tablet copy from peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:29:20.709738  9107 consensus_queue.cc:237] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:20.727098 13298 tablet_bootstrap.cc:492] T f9ae8fac693d4d68b6741f2509e1265e P e0de0545fa0547d3929111e990e6d1b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:20.728627 13298 tablet_bootstrap.cc:492] T f9ae8fac693d4d68b6741f2509e1265e P e0de0545fa0547d3929111e990e6d1b2: Bootstrap complete.
I20240502 21:29:20.727914 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:20.740968 11229 raft_consensus.cc:1261] T 32c306d724954a16be21a34aa4cc0b5f P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:20.741825 13118 consensus_queue.cc:1035] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:20.747293 13009 tablet_bootstrap.cc:492] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:20.747875 13009 tablet_bootstrap.cc:492] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap complete.
I20240502 21:29:20.748253 13009 ts_tablet_manager.cc:1367] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4: Time spent bootstrapping tablet: real 0.747s	user 0.062s	sys 0.013s
I20240502 21:29:20.749245 13009 raft_consensus.cc:348] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:20.749572 13009 raft_consensus.cc:729] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d10e9e156ba74a97a221c85b68ab8bc4, State: Initialized, Role: FOLLOWER
I20240502 21:29:20.749619 13298 ts_tablet_manager.cc:1367] T f9ae8fac693d4d68b6741f2509e1265e P e0de0545fa0547d3929111e990e6d1b2: Time spent bootstrapping tablet: real 0.512s	user 0.054s	sys 0.011s
I20240502 21:29:20.749919 13009 consensus_queue.cc:260] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:20.753731  9242 raft_consensus.cc:1261] T 32c306d724954a16be21a34aa4cc0b5f P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:20.753693  9510 raft_consensus.cc:1261] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:20.754843 13118 consensus_queue.cc:1035] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:20.752487 13298 raft_consensus.cc:348] T f9ae8fac693d4d68b6741f2509e1265e P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:20.759408 13298 raft_consensus.cc:729] T f9ae8fac693d4d68b6741f2509e1265e P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0de0545fa0547d3929111e990e6d1b2, State: Initialized, Role: FOLLOWER
I20240502 21:29:20.759833 13298 consensus_queue.cc:260] T f9ae8fac693d4d68b6741f2509e1265e P e0de0545fa0547d3929111e990e6d1b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
W20240502 21:29:20.761019  9038 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857): Couldn't send request to peer 4cbb5b06a6a44eb6b2145cd538b01af2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32c306d724954a16be21a34aa4cc0b5f. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:20.761502 13118 consensus_queue.cc:1035] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:20.767778  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:29:20.775602  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:20.776093 13298 ts_tablet_manager.cc:1398] T f9ae8fac693d4d68b6741f2509e1265e P e0de0545fa0547d3929111e990e6d1b2: Time spent starting tablet: real 0.026s	user 0.003s	sys 0.000s
I20240502 21:29:20.777423  9399 tablet_copy_service.cc:342] P f2fdd596a8b44bbc848fe40f9f07fbe0: Request end of tablet copy session e0de0545fa0547d3929111e990e6d1b2-f9ae8fac693d4d68b6741f2509e1265e received from {username='slave'} at 127.27.196.72:46235
I20240502 21:29:20.777736  9399 tablet_copy_service.cc:433] P f2fdd596a8b44bbc848fe40f9f07fbe0: ending tablet copy session e0de0545fa0547d3929111e990e6d1b2-f9ae8fac693d4d68b6741f2509e1265e on tablet f9ae8fac693d4d68b6741f2509e1265e with peer e0de0545fa0547d3929111e990e6d1b2
I20240502 21:29:20.796617 13431 tablet_copy_client.cc:783] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:20.797143 13431 tablet_copy_client.cc:647] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:20.812501 13024 raft_consensus.cc:2932] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:20.818612  9241 raft_consensus.cc:2932] T 32c306d724954a16be21a34aa4cc0b5f P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:20.821676 13009 ts_tablet_manager.cc:1398] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4: Time spent starting tablet: real 0.073s	user 0.003s	sys 0.000s
I20240502 21:29:20.835283 13431 tablet_copy_client.cc:523] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:20.837612  8960 catalog_manager.cc:5484] T 32c306d724954a16be21a34aa4cc0b5f P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: config changed from index 4 to 5, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:20.839892  9399 tablet_copy_service.cc:342] P f2fdd596a8b44bbc848fe40f9f07fbe0: Request end of tablet copy session d10e9e156ba74a97a221c85b68ab8bc4-be34ec89876d42b69efe1878c0d9bc19 received from {username='slave'} at 127.27.196.69:60341
I20240502 21:29:20.840101  9399 tablet_copy_service.cc:433] P f2fdd596a8b44bbc848fe40f9f07fbe0: ending tablet copy session d10e9e156ba74a97a221c85b68ab8bc4-be34ec89876d42b69efe1878c0d9bc19 on tablet be34ec89876d42b69efe1878c0d9bc19 with peer d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:29:20.887605  9510 raft_consensus.cc:2932] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:20.893258 13431 tablet_bootstrap.cc:492] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap starting.
I20240502 21:29:20.918418 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:20.910732 11229 raft_consensus.cc:2932] T 32c306d724954a16be21a34aa4cc0b5f P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:20.968412 13454 tablet_copy_client.cc:308] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:43203
I20240502 21:29:20.971740  9399 tablet_copy_service.cc:140] P f2fdd596a8b44bbc848fe40f9f07fbe0: Received BeginTabletCopySession request for tablet 41ee481a3ece47a0905e93c66dbf949d from peer e0de0545fa0547d3929111e990e6d1b2 ({username='slave'} at 127.27.196.72:46235)
I20240502 21:29:20.972007  9399 tablet_copy_service.cc:161] P f2fdd596a8b44bbc848fe40f9f07fbe0: Beginning new tablet copy session on tablet 41ee481a3ece47a0905e93c66dbf949d from peer e0de0545fa0547d3929111e990e6d1b2 at {username='slave'} at 127.27.196.72:46235: session id = e0de0545fa0547d3929111e990e6d1b2-41ee481a3ece47a0905e93c66dbf949d
I20240502 21:29:20.975430  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:20.979450  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:29:20.975046  9399 tablet_copy_source_session.cc:205] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:20.983616 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:20.984140 13454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ee481a3ece47a0905e93c66dbf949d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:20.987048 13298 ts_tablet_manager.cc:902] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2: Initiating tablet copy from peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
I20240502 21:29:20.987581 11094 raft_consensus.cc:1203] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
W20240502 21:29:20.989859  9172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.68:40931, user_credentials={real_user=slave}} blocked reactor thread for 58265.9us
I20240502 21:29:21.010625 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:21.041792  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:21.044271 13454 tablet_copy_client.cc:783] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:21.044991 13454 tablet_copy_client.cc:647] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:21.053781 13454 tablet_copy_client.cc:523] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:21.057300 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:21.089797 13454 tablet_bootstrap.cc:492] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2: Bootstrap starting.
I20240502 21:29:21.159423  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:21.213456 13503 ts_tablet_manager.cc:902] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2: Initiating tablet copy from peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:29:21.220618 13298 tablet_copy_client.cc:308] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.68:40931
I20240502 21:29:21.221992  9533 tablet_copy_service.cc:140] P 19f162e164854f2da623a4bb8f9e6bf6: Received BeginTabletCopySession request for tablet f82c35077224414cb284472d49d7de4f from peer e0de0545fa0547d3929111e990e6d1b2 ({username='slave'} at 127.27.196.72:41441)
I20240502 21:29:21.222194  9533 tablet_copy_service.cc:161] P 19f162e164854f2da623a4bb8f9e6bf6: Beginning new tablet copy session on tablet f82c35077224414cb284472d49d7de4f from peer e0de0545fa0547d3929111e990e6d1b2 at {username='slave'} at 127.27.196.72:41441: session id = e0de0545fa0547d3929111e990e6d1b2-f82c35077224414cb284472d49d7de4f
I20240502 21:29:21.226408  9533 tablet_copy_source_session.cc:205] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:21.246744 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f82c35077224414cb284472d49d7de4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:21.252900 11496 raft_consensus.cc:1203] T f9ae8fac693d4d68b6741f2509e1265e P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:29:21.293522 13298 tablet_copy_client.cc:783] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:21.294003 13298 tablet_copy_client.cc:647] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:21.300681 13298 tablet_copy_client.cc:523] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:21.306882 13298 tablet_bootstrap.cc:492] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2: Bootstrap starting.
I20240502 21:29:21.345309  9107 consensus_queue.cc:237] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:21.352216  9242 raft_consensus.cc:1261] T dc1970b9263d41abb575b9d8639c59ae P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:29:21.352566  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc1970b9263d41abb575b9d8639c59ae. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:21.353309 13468 consensus_queue.cc:1035] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:21.356277  9508 raft_consensus.cc:1261] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:21.357375 13024 consensus_queue.cc:1035] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:21.370436 13503 tablet_copy_client.cc:308] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:42177
I20240502 21:29:21.371670  9128 tablet_copy_service.cc:140] P 5543ddab5b484642b139b7c848932a58: Received BeginTabletCopySession request for tablet 32c306d724954a16be21a34aa4cc0b5f from peer 4cbb5b06a6a44eb6b2145cd538b01af2 ({username='slave'} at 127.27.196.71:58633)
I20240502 21:29:21.371886  9128 tablet_copy_service.cc:161] P 5543ddab5b484642b139b7c848932a58: Beginning new tablet copy session on tablet 32c306d724954a16be21a34aa4cc0b5f from peer 4cbb5b06a6a44eb6b2145cd538b01af2 at {username='slave'} at 127.27.196.71:58633: session id = 4cbb5b06a6a44eb6b2145cd538b01af2-32c306d724954a16be21a34aa4cc0b5f
I20240502 21:29:21.375221  9128 tablet_copy_source_session.cc:205] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:21.377169 13503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32c306d724954a16be21a34aa4cc0b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:21.379491 13431 tablet_bootstrap.cc:492] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:21.380913 13468 raft_consensus.cc:2932] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:21.382624 13431 tablet_bootstrap.cc:492] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap complete.
I20240502 21:29:21.382997 13431 ts_tablet_manager.cc:1367] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2: Time spent bootstrapping tablet: real 0.507s	user 0.050s	sys 0.013s
I20240502 21:29:21.384663 13431 raft_consensus.cc:348] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:21.384339  9241 raft_consensus.cc:2932] T dc1970b9263d41abb575b9d8639c59ae P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:21.384924 13431 raft_consensus.cc:729] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbb5b06a6a44eb6b2145cd538b01af2, State: Initialized, Role: FOLLOWER
I20240502 21:29:21.385228 13431 consensus_queue.cc:260] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:21.394640 13454 tablet_bootstrap.cc:492] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:21.396775 13431 ts_tablet_manager.cc:1398] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.003s
I20240502 21:29:21.398479  9128 tablet_copy_service.cc:342] P 5543ddab5b484642b139b7c848932a58: Request end of tablet copy session 4cbb5b06a6a44eb6b2145cd538b01af2-f0259f01acb249dd99c662151dc21916 received from {username='slave'} at 127.27.196.71:58633
I20240502 21:29:21.398654  9128 tablet_copy_service.cc:433] P 5543ddab5b484642b139b7c848932a58: ending tablet copy session 4cbb5b06a6a44eb6b2145cd538b01af2-f0259f01acb249dd99c662151dc21916 on tablet f0259f01acb249dd99c662151dc21916 with peer 4cbb5b06a6a44eb6b2145cd538b01af2
I20240502 21:29:21.399713  9508 raft_consensus.cc:2932] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:21.409106  8958 catalog_manager.cc:5484] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:21.413805 13454 tablet_bootstrap.cc:492] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2: Bootstrap complete.
I20240502 21:29:21.414892 13454 ts_tablet_manager.cc:1367] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2: Time spent bootstrapping tablet: real 0.325s	user 0.059s	sys 0.001s
I20240502 21:29:21.415796 13454 raft_consensus.cc:348] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:21.416079 13454 raft_consensus.cc:729] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0de0545fa0547d3929111e990e6d1b2, State: Initialized, Role: FOLLOWER
I20240502 21:29:21.423720 11365 raft_consensus.cc:1203] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:29:21.424744 13454 consensus_queue.cc:260] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:21.426877 13454 ts_tablet_manager.cc:1398] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2: Time spent starting tablet: real 0.012s	user 0.002s	sys 0.000s
I20240502 21:29:21.429757  9399 tablet_copy_service.cc:342] P f2fdd596a8b44bbc848fe40f9f07fbe0: Request end of tablet copy session e0de0545fa0547d3929111e990e6d1b2-41ee481a3ece47a0905e93c66dbf949d received from {username='slave'} at 127.27.196.72:46235
I20240502 21:29:21.430027  9399 tablet_copy_service.cc:433] P f2fdd596a8b44bbc848fe40f9f07fbe0: ending tablet copy session e0de0545fa0547d3929111e990e6d1b2-41ee481a3ece47a0905e93c66dbf949d on tablet 41ee481a3ece47a0905e93c66dbf949d with peer e0de0545fa0547d3929111e990e6d1b2
I20240502 21:29:21.449194 13503 tablet_copy_client.cc:783] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:21.450443 13503 tablet_copy_client.cc:647] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:21.453994 13503 tablet_copy_client.cc:523] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:21.473042 13503 tablet_bootstrap.cc:492] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap starting.
I20240502 21:29:21.495949  9376 consensus_queue.cc:237] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:21.510128 11496 raft_consensus.cc:1261] T f9ae8fac693d4d68b6741f2509e1265e P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:21.518101 13335 consensus_queue.cc:1035] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 21:29:21.539222  9311 consensus_peers.cc:485] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Couldn't send request to peer d10e9e156ba74a97a221c85b68ab8bc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9ae8fac693d4d68b6741f2509e1265e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:21.541942  9242 raft_consensus.cc:1261] T f9ae8fac693d4d68b6741f2509e1265e P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:21.542605 13298 tablet_bootstrap.cc:492] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:21.543239 13541 consensus_queue.cc:1035] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:21.547690 13298 tablet_bootstrap.cc:492] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2: Bootstrap complete.
I20240502 21:29:21.548111 13298 ts_tablet_manager.cc:1367] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2: Time spent bootstrapping tablet: real 0.241s	user 0.048s	sys 0.017s
I20240502 21:29:21.549919 13298 raft_consensus.cc:348] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:21.551614 13298 raft_consensus.cc:729] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0de0545fa0547d3929111e990e6d1b2, State: Initialized, Role: FOLLOWER
I20240502 21:29:21.552207 13298 consensus_queue.cc:260] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:21.553997 13298 ts_tablet_manager.cc:1398] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.001s
I20240502 21:29:21.555683  9533 tablet_copy_service.cc:342] P 19f162e164854f2da623a4bb8f9e6bf6: Request end of tablet copy session e0de0545fa0547d3929111e990e6d1b2-f82c35077224414cb284472d49d7de4f received from {username='slave'} at 127.27.196.72:41441
I20240502 21:29:21.555833  9533 tablet_copy_service.cc:433] P 19f162e164854f2da623a4bb8f9e6bf6: ending tablet copy session e0de0545fa0547d3929111e990e6d1b2-f82c35077224414cb284472d49d7de4f on tablet f82c35077224414cb284472d49d7de4f with peer e0de0545fa0547d3929111e990e6d1b2
I20240502 21:29:21.561374  9107 raft_consensus.cc:1261] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:21.562373 13541 consensus_queue.cc:1035] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:21.575098  9107 raft_consensus.cc:2932] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:21.578846 13335 raft_consensus.cc:2932] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:21.612350 11495 raft_consensus.cc:2932] T f9ae8fac693d4d68b6741f2509e1265e P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:21.616052  8960 catalog_manager.cc:5484] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index 4 to 5, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:21.669219 13503 tablet_bootstrap.cc:492] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:21.675913 13503 tablet_bootstrap.cc:492] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap complete.
I20240502 21:29:21.676307 13503 ts_tablet_manager.cc:1367] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2: Time spent bootstrapping tablet: real 0.203s	user 0.064s	sys 0.004s
I20240502 21:29:21.677397 13503 raft_consensus.cc:348] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:21.677721 13503 raft_consensus.cc:729] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbb5b06a6a44eb6b2145cd538b01af2, State: Initialized, Role: FOLLOWER
I20240502 21:29:21.678004 13503 consensus_queue.cc:260] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:21.681897 13503 ts_tablet_manager.cc:1398] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.001s
I20240502 21:29:21.683661  9128 tablet_copy_service.cc:342] P 5543ddab5b484642b139b7c848932a58: Request end of tablet copy session 4cbb5b06a6a44eb6b2145cd538b01af2-32c306d724954a16be21a34aa4cc0b5f received from {username='slave'} at 127.27.196.71:58633
I20240502 21:29:21.683904  9128 tablet_copy_service.cc:433] P 5543ddab5b484642b139b7c848932a58: ending tablet copy session 4cbb5b06a6a44eb6b2145cd538b01af2-32c306d724954a16be21a34aa4cc0b5f on tablet 32c306d724954a16be21a34aa4cc0b5f with peer 4cbb5b06a6a44eb6b2145cd538b01af2
I20240502 21:29:21.696754  9241 raft_consensus.cc:2932] T f9ae8fac693d4d68b6741f2509e1265e P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:21.733532 11495 raft_consensus.cc:1203] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 21:29:21.789077 11365 raft_consensus.cc:1203] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 21:29:21.848539 13577 ts_tablet_manager.cc:902] T dc1970b9263d41abb575b9d8639c59ae P 6ef0beaf2f6941afb6ea9f9f274517fe: Initiating tablet copy from peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:29:21.880813  9242 consensus_queue.cc:237] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:21.882316 13577 tablet_copy_client.cc:308] T dc1970b9263d41abb575b9d8639c59ae P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:42177
I20240502 21:29:21.885715  9128 tablet_copy_service.cc:140] P 5543ddab5b484642b139b7c848932a58: Received BeginTabletCopySession request for tablet dc1970b9263d41abb575b9d8639c59ae from peer 6ef0beaf2f6941afb6ea9f9f274517fe ({username='slave'} at 127.27.196.70:53767)
I20240502 21:29:21.885957  9128 tablet_copy_service.cc:161] P 5543ddab5b484642b139b7c848932a58: Beginning new tablet copy session on tablet dc1970b9263d41abb575b9d8639c59ae from peer 6ef0beaf2f6941afb6ea9f9f274517fe at {username='slave'} at 127.27.196.70:53767: session id = 6ef0beaf2f6941afb6ea9f9f274517fe-dc1970b9263d41abb575b9d8639c59ae
I20240502 21:29:21.887274 11495 raft_consensus.cc:1203] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 21:29:21.889340  9128 tablet_copy_source_session.cc:205] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:21.903556 13577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc1970b9263d41abb575b9d8639c59ae. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 21:29:21.925446  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3caa36212ba444aa88d153ff9f90122e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:21.927480 11094 raft_consensus.cc:1261] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:21.928349 13004 consensus_queue.cc:1035] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:21.938367  9376 raft_consensus.cc:1261] T 3caa36212ba444aa88d153ff9f90122e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:21.941639 13004 consensus_queue.cc:1035] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:21.943946  9508 raft_consensus.cc:1261] T 3caa36212ba444aa88d153ff9f90122e P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:21.944903 12381 consensus_queue.cc:1035] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:22.001085 13577 tablet_copy_client.cc:783] T dc1970b9263d41abb575b9d8639c59ae P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:22.004082 13577 tablet_copy_client.cc:647] T dc1970b9263d41abb575b9d8639c59ae P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:22.005508 12381 raft_consensus.cc:2932] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:22.012379  9376 raft_consensus.cc:2932] T 3caa36212ba444aa88d153ff9f90122e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:22.017143  9510 raft_consensus.cc:2932] T 3caa36212ba444aa88d153ff9f90122e P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:22.023862  8959 catalog_manager.cc:5484] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: config changed from index 4 to 5, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:22.021543 11094 raft_consensus.cc:2932] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:22.026592 13577 tablet_copy_client.cc:523] T dc1970b9263d41abb575b9d8639c59ae P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:22.034840 13577 tablet_bootstrap.cc:492] T dc1970b9263d41abb575b9d8639c59ae P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap starting.
I20240502 21:29:22.063166  9107 consensus_queue.cc:237] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:22.066263  9510 raft_consensus.cc:1261] T 79dedaa327b6479e8325f91481c1688e P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:29:22.066779  9041 consensus_peers.cc:485] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 -> Peer e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467): Couldn't send request to peer e0de0545fa0547d3929111e990e6d1b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79dedaa327b6479e8325f91481c1688e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:22.067066  9376 raft_consensus.cc:1261] T 79dedaa327b6479e8325f91481c1688e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:22.067193 13470 consensus_queue.cc:1035] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:22.067900 13471 consensus_queue.cc:1035] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:22.108146 13471 raft_consensus.cc:2932] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:22.112221  9508 raft_consensus.cc:2932] T 79dedaa327b6479e8325f91481c1688e P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:22.115670  9377 raft_consensus.cc:2932] T 79dedaa327b6479e8325f91481c1688e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:22.121729  8960 catalog_manager.cc:5484] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index -1 to 4, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:22.160619 13638 ts_tablet_manager.cc:902] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4: Initiating tablet copy from peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:29:22.182703 13638 tablet_copy_client.cc:308] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:43203
I20240502 21:29:22.183876  9399 tablet_copy_service.cc:140] P f2fdd596a8b44bbc848fe40f9f07fbe0: Received BeginTabletCopySession request for tablet f9ae8fac693d4d68b6741f2509e1265e from peer d10e9e156ba74a97a221c85b68ab8bc4 ({username='slave'} at 127.27.196.69:60341)
I20240502 21:29:22.184127  9399 tablet_copy_service.cc:161] P f2fdd596a8b44bbc848fe40f9f07fbe0: Beginning new tablet copy session on tablet f9ae8fac693d4d68b6741f2509e1265e from peer d10e9e156ba74a97a221c85b68ab8bc4 at {username='slave'} at 127.27.196.69:60341: session id = d10e9e156ba74a97a221c85b68ab8bc4-f9ae8fac693d4d68b6741f2509e1265e
I20240502 21:29:22.187821  9399 tablet_copy_source_session.cc:205] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:22.189033 13577 tablet_bootstrap.cc:492] T dc1970b9263d41abb575b9d8639c59ae P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:22.189617 13577 tablet_bootstrap.cc:492] T dc1970b9263d41abb575b9d8639c59ae P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap complete.
I20240502 21:29:22.189965 13577 ts_tablet_manager.cc:1367] T dc1970b9263d41abb575b9d8639c59ae P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent bootstrapping tablet: real 0.155s	user 0.067s	sys 0.000s
I20240502 21:29:22.190011 13638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9ae8fac693d4d68b6741f2509e1265e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:22.193181 13577 raft_consensus.cc:348] T dc1970b9263d41abb575b9d8639c59ae P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:22.193472 13577 raft_consensus.cc:729] T dc1970b9263d41abb575b9d8639c59ae P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef0beaf2f6941afb6ea9f9f274517fe, State: Initialized, Role: FOLLOWER
I20240502 21:29:22.197664 13577 consensus_queue.cc:260] T dc1970b9263d41abb575b9d8639c59ae P 6ef0beaf2f6941afb6ea9f9f274517fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:22.199245 13638 tablet_copy_client.cc:783] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:22.200171 13638 tablet_copy_client.cc:647] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:22.202807 13638 tablet_copy_client.cc:523] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:22.206949 13638 tablet_bootstrap.cc:492] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap starting.
I20240502 21:29:22.208714 13577 ts_tablet_manager.cc:1398] T dc1970b9263d41abb575b9d8639c59ae P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent starting tablet: real 0.019s	user 0.008s	sys 0.000s
I20240502 21:29:22.209770  9128 tablet_copy_service.cc:342] P 5543ddab5b484642b139b7c848932a58: Request end of tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-dc1970b9263d41abb575b9d8639c59ae received from {username='slave'} at 127.27.196.70:53767
I20240502 21:29:22.209936  9128 tablet_copy_service.cc:433] P 5543ddab5b484642b139b7c848932a58: ending tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-dc1970b9263d41abb575b9d8639c59ae on tablet dc1970b9263d41abb575b9d8639c59ae with peer 6ef0beaf2f6941afb6ea9f9f274517fe
I20240502 21:29:22.287128  9242 consensus_queue.cc:237] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:22.291566  9508 raft_consensus.cc:1261] T 94d2cdf8ae6941668b5f680c558656e8 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:22.292423 13614 consensus_queue.cc:1035] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:22.298952  9107 raft_consensus.cc:1261] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:22.300046 13614 consensus_queue.cc:1035] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:22.330744 13577 ts_tablet_manager.cc:902] T 3caa36212ba444aa88d153ff9f90122e P 6ef0beaf2f6941afb6ea9f9f274517fe: Initiating tablet copy from peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:29:22.338351 13614 raft_consensus.cc:2932] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:22.345413 13638 tablet_bootstrap.cc:492] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:22.346746  9107 raft_consensus.cc:2932] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:22.347914 13577 tablet_copy_client.cc:308] T 3caa36212ba444aa88d153ff9f90122e P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:39071
I20240502 21:29:22.350571  9263 tablet_copy_service.cc:140] P 525a093ea5bf40dc9bbf9dc39895d670: Received BeginTabletCopySession request for tablet 3caa36212ba444aa88d153ff9f90122e from peer 6ef0beaf2f6941afb6ea9f9f274517fe ({username='slave'} at 127.27.196.70:52575)
I20240502 21:29:22.349124 13638 tablet_bootstrap.cc:492] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap complete.
I20240502 21:29:22.350889  9263 tablet_copy_service.cc:161] P 525a093ea5bf40dc9bbf9dc39895d670: Beginning new tablet copy session on tablet 3caa36212ba444aa88d153ff9f90122e from peer 6ef0beaf2f6941afb6ea9f9f274517fe at {username='slave'} at 127.27.196.70:52575: session id = 6ef0beaf2f6941afb6ea9f9f274517fe-3caa36212ba444aa88d153ff9f90122e
I20240502 21:29:22.351500 13638 ts_tablet_manager.cc:1367] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4: Time spent bootstrapping tablet: real 0.145s	user 0.073s	sys 0.007s
I20240502 21:29:22.352819 13638 raft_consensus.cc:348] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:22.353724 13638 raft_consensus.cc:729] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d10e9e156ba74a97a221c85b68ab8bc4, State: Initialized, Role: FOLLOWER
I20240502 21:29:22.354179 13638 consensus_queue.cc:260] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:22.354615  9263 tablet_copy_source_session.cc:205] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:22.356067 13638 ts_tablet_manager.cc:1398] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 21:29:22.359431  9399 tablet_copy_service.cc:342] P f2fdd596a8b44bbc848fe40f9f07fbe0: Request end of tablet copy session d10e9e156ba74a97a221c85b68ab8bc4-f9ae8fac693d4d68b6741f2509e1265e received from {username='slave'} at 127.27.196.69:60341
I20240502 21:29:22.359647  9399 tablet_copy_service.cc:433] P f2fdd596a8b44bbc848fe40f9f07fbe0: ending tablet copy session d10e9e156ba74a97a221c85b68ab8bc4-f9ae8fac693d4d68b6741f2509e1265e on tablet f9ae8fac693d4d68b6741f2509e1265e with peer d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:29:22.361492 13577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3caa36212ba444aa88d153ff9f90122e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:22.362335  9508 raft_consensus.cc:2932] T 94d2cdf8ae6941668b5f680c558656e8 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:22.367316  8959 catalog_manager.cc:5484] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index -1 to 4, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:22.368428 13577 tablet_copy_client.cc:783] T 3caa36212ba444aa88d153ff9f90122e P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:22.370955 13577 tablet_copy_client.cc:647] T 3caa36212ba444aa88d153ff9f90122e P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 1 WAL segments...
W20240502 21:29:22.373368  9175 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467): Couldn't send request to peer e0de0545fa0547d3929111e990e6d1b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 94d2cdf8ae6941668b5f680c558656e8. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:22.378903 13577 tablet_copy_client.cc:523] T 3caa36212ba444aa88d153ff9f90122e P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:22.385663 13577 tablet_bootstrap.cc:492] T 3caa36212ba444aa88d153ff9f90122e P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap starting.
I20240502 21:29:22.537091 11094 raft_consensus.cc:1203] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 21:29:22.546202 13728 ts_tablet_manager.cc:902] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2: Initiating tablet copy from peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:29:22.591634 13577 tablet_bootstrap.cc:492] T 3caa36212ba444aa88d153ff9f90122e P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:22.593784 13577 tablet_bootstrap.cc:492] T 3caa36212ba444aa88d153ff9f90122e P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap complete.
I20240502 21:29:22.594789 13577 ts_tablet_manager.cc:1367] T 3caa36212ba444aa88d153ff9f90122e P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent bootstrapping tablet: real 0.209s	user 0.066s	sys 0.011s
I20240502 21:29:22.595719 13577 raft_consensus.cc:348] T 3caa36212ba444aa88d153ff9f90122e P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:22.596001 13577 raft_consensus.cc:729] T 3caa36212ba444aa88d153ff9f90122e P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef0beaf2f6941afb6ea9f9f274517fe, State: Initialized, Role: FOLLOWER
I20240502 21:29:22.599510 13577 consensus_queue.cc:260] T 3caa36212ba444aa88d153ff9f90122e P 6ef0beaf2f6941afb6ea9f9f274517fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:22.601897 13577 ts_tablet_manager.cc:1398] T 3caa36212ba444aa88d153ff9f90122e P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.000s
I20240502 21:29:22.605059  9263 tablet_copy_service.cc:342] P 525a093ea5bf40dc9bbf9dc39895d670: Request end of tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-3caa36212ba444aa88d153ff9f90122e received from {username='slave'} at 127.27.196.70:52575
I20240502 21:29:22.605226  9263 tablet_copy_service.cc:433] P 525a093ea5bf40dc9bbf9dc39895d670: ending tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-3caa36212ba444aa88d153ff9f90122e on tablet 3caa36212ba444aa88d153ff9f90122e with peer 6ef0beaf2f6941afb6ea9f9f274517fe
I20240502 21:29:22.628379 13728 tablet_copy_client.cc:308] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:42177
I20240502 21:29:22.629501  9128 tablet_copy_service.cc:140] P 5543ddab5b484642b139b7c848932a58: Received BeginTabletCopySession request for tablet 79dedaa327b6479e8325f91481c1688e from peer e0de0545fa0547d3929111e990e6d1b2 ({username='slave'} at 127.27.196.72:35055)
I20240502 21:29:22.629752  9128 tablet_copy_service.cc:161] P 5543ddab5b484642b139b7c848932a58: Beginning new tablet copy session on tablet 79dedaa327b6479e8325f91481c1688e from peer e0de0545fa0547d3929111e990e6d1b2 at {username='slave'} at 127.27.196.72:35055: session id = e0de0545fa0547d3929111e990e6d1b2-79dedaa327b6479e8325f91481c1688e
I20240502 21:29:22.660303  9128 tablet_copy_source_session.cc:205] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:22.666770 13728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79dedaa327b6479e8325f91481c1688e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:22.675740 13728 tablet_copy_client.cc:783] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:22.676064 13728 tablet_copy_client.cc:647] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:22.677516 11229 raft_consensus.cc:1203] T dc1970b9263d41abb575b9d8639c59ae P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:29:22.705336 13728 tablet_copy_client.cc:523] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:22.712751 13728 tablet_bootstrap.cc:492] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2: Bootstrap starting.
I20240502 21:29:22.718482  9377 consensus_queue.cc:237] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:22.738044  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:22.774690 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:22.782586  9242 raft_consensus.cc:1261] T 4674e376bf7440cda619784491685b44 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:22.785697 13541 consensus_queue.cc:1035] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:22.791271  9108 raft_consensus.cc:1261] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:22.792948 13735 consensus_queue.cc:1035] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:22.870050  9108 consensus_queue.cc:237] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:22.877286 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:22.882576 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 21:29:22.885772  9039 consensus_peers.cc:485] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 -> Peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Couldn't send request to peer d10e9e156ba74a97a221c85b68ab8bc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e46065d94e874293a836bc74241b8e06. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:22.887778  9239 raft_consensus.cc:1261] T e46065d94e874293a836bc74241b8e06 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:22.888728 13468 consensus_queue.cc:1035] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:22.894400  9377 raft_consensus.cc:1261] T e46065d94e874293a836bc74241b8e06 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:22.895846 13468 consensus_queue.cc:1035] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:22.959616 13728 tablet_bootstrap.cc:492] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:22.960328 13728 tablet_bootstrap.cc:492] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2: Bootstrap complete.
I20240502 21:29:22.968169 13771 ts_tablet_manager.cc:902] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2: Initiating tablet copy from peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:29:22.971760 13728 ts_tablet_manager.cc:1367] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2: Time spent bootstrapping tablet: real 0.259s	user 0.067s	sys 0.003s
I20240502 21:29:22.972906 13728 raft_consensus.cc:348] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:22.973223 13728 raft_consensus.cc:729] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0de0545fa0547d3929111e990e6d1b2, State: Initialized, Role: FOLLOWER
I20240502 21:29:22.977376 13728 consensus_queue.cc:260] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:22.981832 13728 ts_tablet_manager.cc:1398] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.001s
I20240502 21:29:22.987805 13541 raft_consensus.cc:2932] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:22.995364  9107 raft_consensus.cc:2932] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:22.999385  8956 catalog_manager.cc:5484] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:23.005565  9128 tablet_copy_service.cc:342] P 5543ddab5b484642b139b7c848932a58: Request end of tablet copy session e0de0545fa0547d3929111e990e6d1b2-79dedaa327b6479e8325f91481c1688e received from {username='slave'} at 127.27.196.72:35055
I20240502 21:29:23.005928  9128 tablet_copy_service.cc:433] P 5543ddab5b484642b139b7c848932a58: ending tablet copy session e0de0545fa0547d3929111e990e6d1b2-79dedaa327b6479e8325f91481c1688e on tablet 79dedaa327b6479e8325f91481c1688e with peer e0de0545fa0547d3929111e990e6d1b2
I20240502 21:29:23.015673  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:23.019327 13468 raft_consensus.cc:2932] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:23.029735  9239 raft_consensus.cc:2932] T e46065d94e874293a836bc74241b8e06 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:23.035359  9377 raft_consensus.cc:2932] T e46065d94e874293a836bc74241b8e06 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:23.034623  9242 raft_consensus.cc:2932] T 4674e376bf7440cda619784491685b44 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:23.038910  8959 catalog_manager.cc:5484] T e46065d94e874293a836bc74241b8e06 P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: config changed from index -1 to 4, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:23.048409 11229 raft_consensus.cc:1203] T 3caa36212ba444aa88d153ff9f90122e P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 21:29:23.077051 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:23.090437 13771 tablet_copy_client.cc:308] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:39071
I20240502 21:29:23.105793 11495 raft_consensus.cc:1203] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:29:23.154280  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 21:29:23.156291  9173 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 91629.3us
W20240502 21:29:23.173563  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4674e376bf7440cda619784491685b44. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:23.184687  9263 tablet_copy_service.cc:140] P 525a093ea5bf40dc9bbf9dc39895d670: Received BeginTabletCopySession request for tablet 94d2cdf8ae6941668b5f680c558656e8 from peer e0de0545fa0547d3929111e990e6d1b2 ({username='slave'} at 127.27.196.72:35369)
I20240502 21:29:23.184918  9263 tablet_copy_service.cc:161] P 525a093ea5bf40dc9bbf9dc39895d670: Beginning new tablet copy session on tablet 94d2cdf8ae6941668b5f680c558656e8 from peer e0de0545fa0547d3929111e990e6d1b2 at {username='slave'} at 127.27.196.72:35369: session id = e0de0545fa0547d3929111e990e6d1b2-94d2cdf8ae6941668b5f680c558656e8
I20240502 21:29:23.185077  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:23.188552  9263 tablet_copy_source_session.cc:205] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:23.191179 13771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94d2cdf8ae6941668b5f680c558656e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:23.237993 13771 tablet_copy_client.cc:783] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:23.238354 13771 tablet_copy_client.cc:647] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:23.255278 13771 tablet_copy_client.cc:523] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:23.296797 13771 tablet_bootstrap.cc:492] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2: Bootstrap starting.
I20240502 21:29:23.309844 13817 ts_tablet_manager.cc:902] T 4674e376bf7440cda619784491685b44 P 6ef0beaf2f6941afb6ea9f9f274517fe: Initiating tablet copy from peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:29:23.376289 13817 tablet_copy_client.cc:308] T 4674e376bf7440cda619784491685b44 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:43203
I20240502 21:29:23.425680 13771 tablet_bootstrap.cc:492] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:23.439457  9399 tablet_copy_service.cc:140] P f2fdd596a8b44bbc848fe40f9f07fbe0: Received BeginTabletCopySession request for tablet 4674e376bf7440cda619784491685b44 from peer 6ef0beaf2f6941afb6ea9f9f274517fe ({username='slave'} at 127.27.196.70:34655)
I20240502 21:29:23.439785  9399 tablet_copy_service.cc:161] P f2fdd596a8b44bbc848fe40f9f07fbe0: Beginning new tablet copy session on tablet 4674e376bf7440cda619784491685b44 from peer 6ef0beaf2f6941afb6ea9f9f274517fe at {username='slave'} at 127.27.196.70:34655: session id = 6ef0beaf2f6941afb6ea9f9f274517fe-4674e376bf7440cda619784491685b44
I20240502 21:29:23.444422  9399 tablet_copy_source_session.cc:205] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:23.455538 13771 tablet_bootstrap.cc:492] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2: Bootstrap complete.
I20240502 21:29:23.455988 13771 ts_tablet_manager.cc:1367] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2: Time spent bootstrapping tablet: real 0.159s	user 0.049s	sys 0.017s
I20240502 21:29:23.456964 13771 raft_consensus.cc:348] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:23.457259 13771 raft_consensus.cc:729] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0de0545fa0547d3929111e990e6d1b2, State: Initialized, Role: FOLLOWER
I20240502 21:29:23.457577 13771 consensus_queue.cc:260] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:23.458180 13771 ts_tablet_manager.cc:1398] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240502 21:29:23.460660  9263 tablet_copy_service.cc:342] P 525a093ea5bf40dc9bbf9dc39895d670: Request end of tablet copy session e0de0545fa0547d3929111e990e6d1b2-94d2cdf8ae6941668b5f680c558656e8 received from {username='slave'} at 127.27.196.72:35369
I20240502 21:29:23.460837  9263 tablet_copy_service.cc:433] P 525a093ea5bf40dc9bbf9dc39895d670: ending tablet copy session e0de0545fa0547d3929111e990e6d1b2-94d2cdf8ae6941668b5f680c558656e8 on tablet 94d2cdf8ae6941668b5f680c558656e8 with peer e0de0545fa0547d3929111e990e6d1b2
I20240502 21:29:23.487267 13817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4674e376bf7440cda619784491685b44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:23.506953 13817 tablet_copy_client.cc:783] T 4674e376bf7440cda619784491685b44 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:23.508047 13817 tablet_copy_client.cc:647] T 4674e376bf7440cda619784491685b44 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:23.512282 13817 tablet_copy_client.cc:523] T 4674e376bf7440cda619784491685b44 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:23.518035 13817 tablet_bootstrap.cc:492] T 4674e376bf7440cda619784491685b44 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap starting.
I20240502 21:29:23.545878 13856 ts_tablet_manager.cc:902] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4: Initiating tablet copy from peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:29:23.605393  9510 consensus_queue.cc:237] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:23.612118  9242 raft_consensus.cc:1261] T c56ac926025e47218e9df4088184a278 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:23.614784 13865 consensus_queue.cc:1035] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:23.616632  9107 raft_consensus.cc:1261] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:29:23.618552  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c56ac926025e47218e9df4088184a278. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:23.640595 13865 consensus_queue.cc:1035] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:23.644043 13817 tablet_bootstrap.cc:492] T 4674e376bf7440cda619784491685b44 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:23.645613 13817 tablet_bootstrap.cc:492] T 4674e376bf7440cda619784491685b44 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap complete.
I20240502 21:29:23.646057 13817 ts_tablet_manager.cc:1367] T 4674e376bf7440cda619784491685b44 P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent bootstrapping tablet: real 0.128s	user 0.049s	sys 0.012s
I20240502 21:29:23.646993 13817 raft_consensus.cc:348] T 4674e376bf7440cda619784491685b44 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:23.647282 13817 raft_consensus.cc:729] T 4674e376bf7440cda619784491685b44 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef0beaf2f6941afb6ea9f9f274517fe, State: Initialized, Role: FOLLOWER
I20240502 21:29:23.647617 13817 consensus_queue.cc:260] T 4674e376bf7440cda619784491685b44 P 6ef0beaf2f6941afb6ea9f9f274517fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:23.655138 13817 ts_tablet_manager.cc:1398] T 4674e376bf7440cda619784491685b44 P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20240502 21:29:23.656724  9399 tablet_copy_service.cc:342] P f2fdd596a8b44bbc848fe40f9f07fbe0: Request end of tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-4674e376bf7440cda619784491685b44 received from {username='slave'} at 127.27.196.70:34655
I20240502 21:29:23.656908  9399 tablet_copy_service.cc:433] P f2fdd596a8b44bbc848fe40f9f07fbe0: ending tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-4674e376bf7440cda619784491685b44 on tablet 4674e376bf7440cda619784491685b44 with peer 6ef0beaf2f6941afb6ea9f9f274517fe
I20240502 21:29:23.680688 13856 tablet_copy_client.cc:308] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:42177
I20240502 21:29:23.681469  9128 tablet_copy_service.cc:140] P 5543ddab5b484642b139b7c848932a58: Received BeginTabletCopySession request for tablet e46065d94e874293a836bc74241b8e06 from peer d10e9e156ba74a97a221c85b68ab8bc4 ({username='slave'} at 127.27.196.69:40153)
I20240502 21:29:23.681681  9128 tablet_copy_service.cc:161] P 5543ddab5b484642b139b7c848932a58: Beginning new tablet copy session on tablet e46065d94e874293a836bc74241b8e06 from peer d10e9e156ba74a97a221c85b68ab8bc4 at {username='slave'} at 127.27.196.69:40153: session id = d10e9e156ba74a97a221c85b68ab8bc4-e46065d94e874293a836bc74241b8e06
I20240502 21:29:23.685235  9128 tablet_copy_source_session.cc:205] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:23.685802 13865 raft_consensus.cc:2932] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:23.687072 13856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e46065d94e874293a836bc74241b8e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:23.691980  9107 raft_consensus.cc:2932] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:23.704591  8958 catalog_manager.cc:5484] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New cstate: current_term: 1 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:23.715236  9242 raft_consensus.cc:2932] T c56ac926025e47218e9df4088184a278 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:23.719305 13856 tablet_copy_client.cc:783] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:23.719619 13856 tablet_copy_client.cc:647] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:23.724140 13856 tablet_copy_client.cc:523] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:23.730466 13856 tablet_bootstrap.cc:492] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap starting.
I20240502 21:29:23.746796 11495 raft_consensus.cc:1203] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:29:23.797639 11228 raft_consensus.cc:1203] T 4674e376bf7440cda619784491685b44 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:29:23.898717  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:23.943933 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:23.956887 13856 tablet_bootstrap.cc:492] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:23.982594 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:23.987633 13856 tablet_bootstrap.cc:492] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap complete.
I20240502 21:29:23.988227 13856 ts_tablet_manager.cc:1367] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4: Time spent bootstrapping tablet: real 0.258s	user 0.066s	sys 0.005s
I20240502 21:29:23.989378 13856 raft_consensus.cc:348] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:23.989730 13856 raft_consensus.cc:729] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d10e9e156ba74a97a221c85b68ab8bc4, State: Initialized, Role: FOLLOWER
I20240502 21:29:23.990036 13856 consensus_queue.cc:260] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:23.990789 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:24.018198 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:24.086251  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:29:24.146730 13856 ts_tablet_manager.cc:1398] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4: Time spent starting tablet: real 0.158s	user 0.037s	sys 0.040s
I20240502 21:29:24.147661  9128 tablet_copy_service.cc:342] P 5543ddab5b484642b139b7c848932a58: Request end of tablet copy session d10e9e156ba74a97a221c85b68ab8bc4-e46065d94e874293a836bc74241b8e06 received from {username='slave'} at 127.27.196.69:40153
I20240502 21:29:24.147825  9128 tablet_copy_service.cc:433] P 5543ddab5b484642b139b7c848932a58: ending tablet copy session d10e9e156ba74a97a221c85b68ab8bc4-e46065d94e874293a836bc74241b8e06 on tablet e46065d94e874293a836bc74241b8e06 with peer d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:29:24.154301 13817 ts_tablet_manager.cc:902] T c56ac926025e47218e9df4088184a278 P 6ef0beaf2f6941afb6ea9f9f274517fe: Initiating tablet copy from peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
I20240502 21:29:24.202767  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:24.207379 13817 tablet_copy_client.cc:308] T c56ac926025e47218e9df4088184a278 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.68:40931
I20240502 21:29:24.208500  9533 tablet_copy_service.cc:140] P 19f162e164854f2da623a4bb8f9e6bf6: Received BeginTabletCopySession request for tablet c56ac926025e47218e9df4088184a278 from peer 6ef0beaf2f6941afb6ea9f9f274517fe ({username='slave'} at 127.27.196.70:53371)
I20240502 21:29:24.208709  9533 tablet_copy_service.cc:161] P 19f162e164854f2da623a4bb8f9e6bf6: Beginning new tablet copy session on tablet c56ac926025e47218e9df4088184a278 from peer 6ef0beaf2f6941afb6ea9f9f274517fe at {username='slave'} at 127.27.196.70:53371: session id = 6ef0beaf2f6941afb6ea9f9f274517fe-c56ac926025e47218e9df4088184a278
I20240502 21:29:24.212222  9533 tablet_copy_source_session.cc:205] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:24.214689 13817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c56ac926025e47218e9df4088184a278. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:24.269287  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:24.327260 13817 tablet_copy_client.cc:783] T c56ac926025e47218e9df4088184a278 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:24.327724 13817 tablet_copy_client.cc:647] T c56ac926025e47218e9df4088184a278 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:24.341260 13817 tablet_copy_client.cc:523] T c56ac926025e47218e9df4088184a278 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:24.353467 13817 tablet_bootstrap.cc:492] T c56ac926025e47218e9df4088184a278 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap starting.
I20240502 21:29:24.422662  9107 consensus_queue.cc:237] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:24.425616 11094 raft_consensus.cc:1203] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
W20240502 21:29:24.457561  9039 consensus_peers.cc:485] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 -> Peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Couldn't send request to peer d10e9e156ba74a97a221c85b68ab8bc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79dedaa327b6479e8325f91481c1688e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:24.458412 11495 raft_consensus.cc:1261] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:24.458554  9378 raft_consensus.cc:1261] T 79dedaa327b6479e8325f91481c1688e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:24.460726 13470 consensus_queue.cc:1035] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:24.460927  9510 raft_consensus.cc:1261] T 79dedaa327b6479e8325f91481c1688e P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:24.461570 13470 consensus_queue.cc:1035] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:24.462213 13949 consensus_queue.cc:1035] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 21:29:24.535748 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:24.542428  9508 raft_consensus.cc:2932] T 79dedaa327b6479e8325f91481c1688e P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:24.545960  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:24.539026 13976 raft_consensus.cc:2932] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:24.578351  9377 raft_consensus.cc:2932] T 79dedaa327b6479e8325f91481c1688e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:24.590005  8962 catalog_manager.cc:5484] T 79dedaa327b6479e8325f91481c1688e P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: config changed from index 4 to 5, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:24.599395 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:24.673941  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:24.673734 11495 raft_consensus.cc:2932] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:24.689607 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:24.706820  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:29:24.851521 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:24.920890  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:24.975695  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:25.025221  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:25.030680 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:25.051966 14032 ts_tablet_manager.cc:902] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4: Initiating tablet copy from peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:29:25.116873 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:25.120415 13817 tablet_bootstrap.cc:492] T c56ac926025e47218e9df4088184a278 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:25.121007 13817 tablet_bootstrap.cc:492] T c56ac926025e47218e9df4088184a278 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap complete.
I20240502 21:29:25.121338 13817 ts_tablet_manager.cc:1367] T c56ac926025e47218e9df4088184a278 P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent bootstrapping tablet: real 0.768s	user 0.081s	sys 0.025s
I20240502 21:29:25.122449 13817 raft_consensus.cc:348] T c56ac926025e47218e9df4088184a278 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:25.122711 13817 raft_consensus.cc:729] T c56ac926025e47218e9df4088184a278 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef0beaf2f6941afb6ea9f9f274517fe, State: Initialized, Role: FOLLOWER
I20240502 21:29:25.123020 13817 consensus_queue.cc:260] T c56ac926025e47218e9df4088184a278 P 6ef0beaf2f6941afb6ea9f9f274517fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:25.138686  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:25.157967 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:25.173066  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:29:25.184552 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:25.232607 13817 ts_tablet_manager.cc:1398] T c56ac926025e47218e9df4088184a278 P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent starting tablet: real 0.111s	user 0.010s	sys 0.013s
I20240502 21:29:25.233597  9533 tablet_copy_service.cc:342] P 19f162e164854f2da623a4bb8f9e6bf6: Request end of tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-c56ac926025e47218e9df4088184a278 received from {username='slave'} at 127.27.196.70:53371
I20240502 21:29:25.233775  9533 tablet_copy_service.cc:433] P 19f162e164854f2da623a4bb8f9e6bf6: ending tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-c56ac926025e47218e9df4088184a278 on tablet c56ac926025e47218e9df4088184a278 with peer 6ef0beaf2f6941afb6ea9f9f274517fe
I20240502 21:29:25.322983 14032 tablet_copy_client.cc:308] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:42177
I20240502 21:29:25.324967  9128 tablet_copy_service.cc:140] P 5543ddab5b484642b139b7c848932a58: Received BeginTabletCopySession request for tablet 79dedaa327b6479e8325f91481c1688e from peer d10e9e156ba74a97a221c85b68ab8bc4 ({username='slave'} at 127.27.196.69:40153)
I20240502 21:29:25.325204  9128 tablet_copy_service.cc:161] P 5543ddab5b484642b139b7c848932a58: Beginning new tablet copy session on tablet 79dedaa327b6479e8325f91481c1688e from peer d10e9e156ba74a97a221c85b68ab8bc4 at {username='slave'} at 127.27.196.69:40153: session id = d10e9e156ba74a97a221c85b68ab8bc4-79dedaa327b6479e8325f91481c1688e
I20240502 21:29:25.329773  9128 tablet_copy_source_session.cc:205] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:25.332500 14032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79dedaa327b6479e8325f91481c1688e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:25.346375 14032 tablet_copy_client.cc:783] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:25.368575 14032 tablet_copy_client.cc:647] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:25.372627 14032 tablet_copy_client.cc:523] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:25.379830 14032 tablet_bootstrap.cc:492] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap starting.
I20240502 21:29:25.472882 11229 raft_consensus.cc:1203] T c56ac926025e47218e9df4088184a278 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
W20240502 21:29:25.522468  9445 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 48711.8us
I20240502 21:29:25.543080 14032 tablet_bootstrap.cc:492] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:25.544831 14032 tablet_bootstrap.cc:492] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap complete.
I20240502 21:29:25.545423 14032 ts_tablet_manager.cc:1367] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4: Time spent bootstrapping tablet: real 0.166s	user 0.102s	sys 0.032s
I20240502 21:29:25.547228 14032 raft_consensus.cc:348] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:25.547660 14032 raft_consensus.cc:729] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d10e9e156ba74a97a221c85b68ab8bc4, State: Initialized, Role: FOLLOWER
I20240502 21:29:25.548146 14032 consensus_queue.cc:260] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:25.557615 14032 ts_tablet_manager.cc:1398] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.003s
I20240502 21:29:25.558760  9128 tablet_copy_service.cc:342] P 5543ddab5b484642b139b7c848932a58: Request end of tablet copy session d10e9e156ba74a97a221c85b68ab8bc4-79dedaa327b6479e8325f91481c1688e received from {username='slave'} at 127.27.196.69:40153
I20240502 21:29:25.558993  9128 tablet_copy_service.cc:433] P 5543ddab5b484642b139b7c848932a58: ending tablet copy session d10e9e156ba74a97a221c85b68ab8bc4-79dedaa327b6479e8325f91481c1688e on tablet 79dedaa327b6479e8325f91481c1688e with peer d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:29:25.984526 11093 raft_consensus.cc:1203] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 21:29:26.120195  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:26.143087  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:26.161181  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:26.197463 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:26.237613  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:26.239661  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:29:26.268049  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 21:29:26.272182  9174 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.67:43203, user_credentials={real_user=slave}} blocked reactor thread for 91777.4us
I20240502 21:29:26.295708 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:26.302310 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:26.308195 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:26.321478 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:26.348176 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:26.397727  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 21:29:26.405181  9313 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.72:40467, user_credentials={real_user=slave}} blocked reactor thread for 141919us
I20240502 21:29:26.468541 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:26.484620  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:26.493300 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:26.559414  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:26.566198 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:26.625285  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:26.692432 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:26.723452  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:29:26.762822  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:26.788107 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:26.788902 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:27.240037  9239 consensus_queue.cc:237] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:27.290402 11094 raft_consensus.cc:1261] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:27.293982  9508 raft_consensus.cc:1261] T 51936fb1f678458a8795f8246590addb P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:27.296605 14236 consensus_queue.cc:1035] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:27.307756 14236 consensus_queue.cc:1035] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:27.384349 14236 raft_consensus.cc:2932] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:27.388159  9510 raft_consensus.cc:2932] T 51936fb1f678458a8795f8246590addb P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:27.401309  8956 catalog_manager.cc:5484] T 51936fb1f678458a8795f8246590addb P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: config changed from index 4 to 5, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:27.429416 11094 raft_consensus.cc:2932] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:27.456166  9085 tablet_service.cc:1514] Processing DeleteTablet for tablet 51936fb1f678458a8795f8246590addb with delete_type TABLET_DATA_TOMBSTONED (TS 5543ddab5b484642b139b7c848932a58 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47814
I20240502 21:29:27.482748 14356 tablet_replica.cc:330] T 51936fb1f678458a8795f8246590addb P 5543ddab5b484642b139b7c848932a58: stopping tablet replica
I20240502 21:29:27.484951 14356 raft_consensus.cc:2229] T 51936fb1f678458a8795f8246590addb P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:27.485517 14356 raft_consensus.cc:2258] T 51936fb1f678458a8795f8246590addb P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:27.487826 14356 ts_tablet_manager.cc:1875] T 51936fb1f678458a8795f8246590addb P 5543ddab5b484642b139b7c848932a58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:27.533954 14356 ts_tablet_manager.cc:1888] T 51936fb1f678458a8795f8246590addb P 5543ddab5b484642b139b7c848932a58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:29:27.535228 14356 log.cc:1196] T 51936fb1f678458a8795f8246590addb P 5543ddab5b484642b139b7c848932a58: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/wals/51936fb1f678458a8795f8246590addb
I20240502 21:29:27.537043  8945 catalog_manager.cc:4830] TS 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): tablet 51936fb1f678458a8795f8246590addb (table rebalance_test_table_2 [id=c0a0ef39a2a246449eb582cc1d993484]) successfully deleted
I20240502 21:29:27.585392  9508 consensus_queue.cc:237] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:27.619714 11229 raft_consensus.cc:1261] T 3d1a5050d3f944969a560994a23700ad P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:27.620440 14272 consensus_queue.cc:1035] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:27.628787  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:27.636197  9107 raft_consensus.cc:1261] T 3d1a5050d3f944969a560994a23700ad P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:27.639205 14272 consensus_queue.cc:1035] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:27.677126 14090 raft_consensus.cc:2932] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:27.681823 11228 raft_consensus.cc:2932] T 3d1a5050d3f944969a560994a23700ad P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:27.704283  8963 catalog_manager.cc:5484] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: config changed from index 4 to 5, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New cstate: current_term: 1 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:27.715512  9221 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d1a5050d3f944969a560994a23700ad with delete_type TABLET_DATA_TOMBSTONED (TS 525a093ea5bf40dc9bbf9dc39895d670 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44408
I20240502 21:29:27.722146 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:27.734728  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:27.737756  9107 raft_consensus.cc:2932] T 3d1a5050d3f944969a560994a23700ad P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:27.751839 14409 tablet_replica.cc:330] T 3d1a5050d3f944969a560994a23700ad P 525a093ea5bf40dc9bbf9dc39895d670: stopping tablet replica
I20240502 21:29:27.752892 14409 raft_consensus.cc:2229] T 3d1a5050d3f944969a560994a23700ad P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:27.771772 14409 raft_consensus.cc:2258] T 3d1a5050d3f944969a560994a23700ad P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:27.790455  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:29:27.826709 14409 ts_tablet_manager.cc:1875] T 3d1a5050d3f944969a560994a23700ad P 525a093ea5bf40dc9bbf9dc39895d670: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:27.830775 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:27.839358  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:27.869836 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:27.885160  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:27.952759 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:27.963605 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:27.993714  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:28.020622 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:28.090478 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:28.110730 14409 ts_tablet_manager.cc:1888] T 3d1a5050d3f944969a560994a23700ad P 525a093ea5bf40dc9bbf9dc39895d670: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:29:28.128713 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:28.150704 14409 log.cc:1196] T 3d1a5050d3f944969a560994a23700ad P 525a093ea5bf40dc9bbf9dc39895d670: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/wals/3d1a5050d3f944969a560994a23700ad
I20240502 21:29:28.152390  8949 catalog_manager.cc:4830] TS 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): tablet 3d1a5050d3f944969a560994a23700ad (table rebalance_test_table_9 [id=16e8100140034581b2a773dcf740998d]) successfully deleted
I20240502 21:29:28.161846  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:28.168764  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:29:28.510557  9107 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f0259f01acb249dd99c662151dc21916"
dest_uuid: "5543ddab5b484642b139b7c848932a58"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:46292
I20240502 21:29:28.511000  9107 raft_consensus.cc:595] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:29:28.645632  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:28.684571 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:28.784197  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:28.843434  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:28.882512  9085 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:29:28.949590 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:28.951097 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:28.962780 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:28.978272  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:28.997236 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:29.009025 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:29.041913 13470 raft_consensus.cc:979] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58: : Instructing follower f2fdd596a8b44bbc848fe40f9f07fbe0 to start an election
I20240502 21:29:29.042145 13470 raft_consensus.cc:1067] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Signalling peer f2fdd596a8b44bbc848fe40f9f07fbe0 to start an election
I20240502 21:29:29.043493  9378 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f0259f01acb249dd99c662151dc21916"
dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
 from {username='slave'} at 127.27.196.65:38501
I20240502 21:29:29.043843  9378 raft_consensus.cc:482] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:29.043988  9378 raft_consensus.cc:3037] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:29.048148  9378 raft_consensus.cc:504] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:29.050154  9508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0259f01acb249dd99c662151dc21916" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6"
I20240502 21:29:29.050499  9508 raft_consensus.cc:3037] T f0259f01acb249dd99c662151dc21916 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:29.057442  9508 raft_consensus.cc:2454] T f0259f01acb249dd99c662151dc21916 P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 2.
I20240502 21:29:29.060180  9105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0259f01acb249dd99c662151dc21916" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5543ddab5b484642b139b7c848932a58"
I20240502 21:29:29.060511  9105 raft_consensus.cc:3032] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:29.060631  9105 raft_consensus.cc:729] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Running, Role: LEADER
I20240502 21:29:29.061110  9105 consensus_queue.cc:260] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:29.061581  9105 raft_consensus.cc:3037] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:29.069509  9105 raft_consensus.cc:2454] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 2.
I20240502 21:29:29.070255  9310 leader_election.cc:302] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 5543ddab5b484642b139b7c848932a58, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:29:29.070834 14425 raft_consensus.cc:2781] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:29.071079 14425 raft_consensus.cc:686] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 LEADER]: Becoming Leader. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Running, Role: LEADER
I20240502 21:29:29.071516 14425 consensus_queue.cc:237] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:29.074848 11364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0259f01acb249dd99c662151dc21916" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2"
I20240502 21:29:29.075327 11364 raft_consensus.cc:3037] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:29.076776  9378 leader_election.cc:288] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 2 election: Requested vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857)
I20240502 21:29:29.078912 11364 raft_consensus.cc:2454] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 2.
I20240502 21:29:29.090652  8956 catalog_manager.cc:5484] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: term changed from 1 to 2, leader changed from 5543ddab5b484642b139b7c848932a58 (127.27.196.65) to f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67). New cstate: current_term: 2 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:29.097510 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:29.116803 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 21:29:29.117040  9445 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 137495us
I20240502 21:29:29.121709  9356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:29.150573  9356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:29.171473  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:29:29.589268  9242 consensus_queue.cc:237] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:29.598718 11364 raft_consensus.cc:1261] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:29:29.599588 14425 consensus_queue.cc:1035] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:29.601989 11496 raft_consensus.cc:1261] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:29.603919  9105 raft_consensus.cc:1261] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:29.605839 14345 consensus_queue.cc:1035] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:29.613075 14236 consensus_queue.cc:1035] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:29.617974 14345 raft_consensus.cc:2932] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:29.624512 11496 raft_consensus.cc:2932] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:29.652747  9508 raft_consensus.cc:1261] T f0259f01acb249dd99c662151dc21916 P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:29:29.655134 14425 consensus_queue.cc:1035] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 21:29:29.664467  9105 raft_consensus.cc:1261] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20240502 21:29:29.664361  9172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.68:40931, user_credentials={real_user=slave}} blocked reactor thread for 51979.3us
I20240502 21:29:29.676100 14425 consensus_queue.cc:1035] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:29.697862  8962 catalog_manager.cc:5484] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 reported cstate change: config changed from index 4 to 5, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:29.727533  9107 raft_consensus.cc:2932] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:29.782020 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:29.788709  9490 tablet_service.cc:1514] Processing DeleteTablet for tablet 94d2cdf8ae6941668b5f680c558656e8 with delete_type TABLET_DATA_TOMBSTONED (TS 19f162e164854f2da623a4bb8f9e6bf6 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54670
I20240502 21:29:29.820950 14653 tablet_replica.cc:330] T 94d2cdf8ae6941668b5f680c558656e8 P 19f162e164854f2da623a4bb8f9e6bf6: stopping tablet replica
I20240502 21:29:29.824486 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:29.828519 14653 raft_consensus.cc:2229] T 94d2cdf8ae6941668b5f680c558656e8 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:29.829078 14653 raft_consensus.cc:2258] T 94d2cdf8ae6941668b5f680c558656e8 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:29.864532  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:29.876576  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:29.877768  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:29.880807 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:29.888970 14653 ts_tablet_manager.cc:1875] T 94d2cdf8ae6941668b5f680c558656e8 P 19f162e164854f2da623a4bb8f9e6bf6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:29.947618  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:29.949357 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:29.970182  9242 consensus_queue.cc:237] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:29:29.972636  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:29:29.981971  9356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:30.016296 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:30.039650  9378 raft_consensus.cc:1261] T 4d1a31513bb64578a85adf1f622e5bc0 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:30.040851 14617 consensus_queue.cc:1035] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:30.068148  9107 raft_consensus.cc:1261] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:30.069419 14617 consensus_queue.cc:1035] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:30.090559 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:30.111276 14345 raft_consensus.cc:2932] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } }
I20240502 21:29:30.125944  8963 catalog_manager.cc:5484] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: config changed from index 4 to 5, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } }
I20240502 21:29:30.130549  9107 raft_consensus.cc:2932] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } }
I20240502 21:29:30.150336 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:30.177520  9378 raft_consensus.cc:2932] T 4d1a31513bb64578a85adf1f622e5bc0 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } }
I20240502 21:29:30.192081 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:30.196033  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:29:30.199499 14653 ts_tablet_manager.cc:1888] T 94d2cdf8ae6941668b5f680c558656e8 P 19f162e164854f2da623a4bb8f9e6bf6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:29:30.199734 14653 log.cc:1196] T 94d2cdf8ae6941668b5f680c558656e8 P 19f162e164854f2da623a4bb8f9e6bf6: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/wals/94d2cdf8ae6941668b5f680c558656e8
I20240502 21:29:30.200935  8945 catalog_manager.cc:4830] TS 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): tablet 94d2cdf8ae6941668b5f680c558656e8 (table rebalance_test_table_7 [id=ef412fa8fe9249729cb99879a5c8343a]) successfully deleted
I20240502 21:29:30.248351 11209 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d1a31513bb64578a85adf1f622e5bc0 with delete_type TABLET_DATA_TOMBSTONED (TS 6ef0beaf2f6941afb6ea9f9f274517fe not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36872
I20240502 21:29:30.252465  9356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:30.276304 14696 tablet_replica.cc:330] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: stopping tablet replica
I20240502 21:29:30.278347 14696 raft_consensus.cc:2229] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:30.278941 14696 raft_consensus.cc:2258] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:30.280704 14696 ts_tablet_manager.cc:1875] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:30.332238 14696 ts_tablet_manager.cc:1888] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:29:30.332499 14696 log.cc:1196] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal/wals/4d1a31513bb64578a85adf1f622e5bc0
I20240502 21:29:30.333786  8946 catalog_manager.cc:4830] TS 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): tablet 4d1a31513bb64578a85adf1f622e5bc0 (table rebalance_test_table_1 [id=f52c6c3804274777abfa00bb33ae9b16]) successfully deleted
I20240502 21:29:30.429946  9507 consensus_queue.cc:237] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:30.459494 11496 raft_consensus.cc:1261] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:30.459791  9378 raft_consensus.cc:1261] T 1497b781f36c43b39d22a05bb524c59d P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:30.461374 14090 consensus_queue.cc:1035] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:30.462129 14090 consensus_queue.cc:1035] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:30.491161 14090 raft_consensus.cc:2932] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:30.495029  9377 raft_consensus.cc:2932] T 1497b781f36c43b39d22a05bb524c59d P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:30.514267  8963 catalog_manager.cc:5484] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: config changed from index 4 to 5, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New cstate: current_term: 1 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:30.521378  9221 tablet_service.cc:1514] Processing DeleteTablet for tablet 1497b781f36c43b39d22a05bb524c59d with delete_type TABLET_DATA_TOMBSTONED (TS 525a093ea5bf40dc9bbf9dc39895d670 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44408
I20240502 21:29:30.522225 14725 tablet_replica.cc:330] T 1497b781f36c43b39d22a05bb524c59d P 525a093ea5bf40dc9bbf9dc39895d670: stopping tablet replica
I20240502 21:29:30.541580 11495 raft_consensus.cc:2932] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:30.539693 14725 raft_consensus.cc:2229] T 1497b781f36c43b39d22a05bb524c59d P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:30.545018 14725 raft_consensus.cc:2258] T 1497b781f36c43b39d22a05bb524c59d P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:30.546294 14725 ts_tablet_manager.cc:1875] T 1497b781f36c43b39d22a05bb524c59d P 525a093ea5bf40dc9bbf9dc39895d670: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:30.613001 14725 ts_tablet_manager.cc:1888] T 1497b781f36c43b39d22a05bb524c59d P 525a093ea5bf40dc9bbf9dc39895d670: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:29:30.614638 14725 log.cc:1196] T 1497b781f36c43b39d22a05bb524c59d P 525a093ea5bf40dc9bbf9dc39895d670: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/wals/1497b781f36c43b39d22a05bb524c59d
I20240502 21:29:30.617096  8949 catalog_manager.cc:4830] TS 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): tablet 1497b781f36c43b39d22a05bb524c59d (table rebalance_test_table_3 [id=a237d7bda970485480e98020c6ac738f]) successfully deleted
I20240502 21:29:30.888697 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:30.899194 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:30.904383 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:30.937247  9507 consensus_queue.cc:237] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:30.938469  9356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:30.940441 11495 raft_consensus.cc:1261] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:30.947214 14666 consensus_queue.cc:1035] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:30.947510  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:30.951503 14670 raft_consensus.cc:2932] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:30.956362 11495 raft_consensus.cc:2932] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:30.965584  8962 catalog_manager.cc:5484] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 reported cstate change: config changed from index 5 to 6, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New cstate: current_term: 1 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:31.003872  9356 tablet_service.cc:1514] Processing DeleteTablet for tablet 1497b781f36c43b39d22a05bb524c59d with delete_type TABLET_DATA_TOMBSTONED (TS f2fdd596a8b44bbc848fe40f9f07fbe0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55106
I20240502 21:29:31.006142 14824 tablet_replica.cc:330] T 1497b781f36c43b39d22a05bb524c59d P f2fdd596a8b44bbc848fe40f9f07fbe0: stopping tablet replica
I20240502 21:29:31.007936 14824 raft_consensus.cc:2229] T 1497b781f36c43b39d22a05bb524c59d P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:31.010555 14824 raft_consensus.cc:2258] T 1497b781f36c43b39d22a05bb524c59d P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:31.018054 14824 ts_tablet_manager.cc:1875] T 1497b781f36c43b39d22a05bb524c59d P f2fdd596a8b44bbc848fe40f9f07fbe0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:31.029352  9507 consensus_queue.cc:237] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:31.033401 11496 raft_consensus.cc:1261] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:31.036494 14666 consensus_queue.cc:1035] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 21:29:31.037314  9444 consensus_peers.cc:485] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857): Couldn't send request to peer 4cbb5b06a6a44eb6b2145cd538b01af2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1497b781f36c43b39d22a05bb524c59d. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:31.041983 14666 raft_consensus.cc:2932] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:31.047375 11495 raft_consensus.cc:2932] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:31.052610  8945 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 1497b781f36c43b39d22a05bb524c59d with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 21:29:31.056097  8963 catalog_manager.cc:5484] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: config changed from index 6 to 7, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New cstate: current_term: 1 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:31.066154 14824 ts_tablet_manager.cc:1888] T 1497b781f36c43b39d22a05bb524c59d P f2fdd596a8b44bbc848fe40f9f07fbe0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:29:31.070046 14824 log.cc:1196] T 1497b781f36c43b39d22a05bb524c59d P f2fdd596a8b44bbc848fe40f9f07fbe0: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/wals/1497b781f36c43b39d22a05bb524c59d
I20240502 21:29:31.084300  8948 catalog_manager.cc:4830] TS f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): tablet 1497b781f36c43b39d22a05bb524c59d (table rebalance_test_table_3 [id=a237d7bda970485480e98020c6ac738f]) successfully deleted
W20240502 21:29:31.192440  9038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.68:40931, user_credentials={real_user=slave}} blocked reactor thread for 398239us
W20240502 21:29:31.233062  9313 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.66:39071, user_credentials={real_user=slave}} blocked reactor thread for 73645.3us
I20240502 21:29:31.262133 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:31.284163  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:31.290145  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:31.325764 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:31.346318 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:31.409438 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:31.435585  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:29:31.456089  9310 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.65:42177, user_credentials={real_user=slave}} blocked reactor thread for 211697us
I20240502 21:29:31.546671 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 21:29:31.598891  9311 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.69:42049, user_credentials={real_user=slave}} blocked reactor thread for 64481us
W20240502 21:29:31.629125  9310 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.65:42177, user_credentials={real_user=slave}} blocked reactor thread for 75570.1us
I20240502 21:29:31.685279  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:29:31.691715 14877 ts_tablet_manager.cc:902] T 1497b781f36c43b39d22a05bb524c59d P 4cbb5b06a6a44eb6b2145cd538b01af2: Initiating tablet copy from peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
I20240502 21:29:31.709712  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:29:31.785012  9356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:31.795595 14877 tablet_copy_client.cc:308] T 1497b781f36c43b39d22a05bb524c59d P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.68:40931
I20240502 21:29:31.795713 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:31.796871  9533 tablet_copy_service.cc:140] P 19f162e164854f2da623a4bb8f9e6bf6: Received BeginTabletCopySession request for tablet 1497b781f36c43b39d22a05bb524c59d from peer 4cbb5b06a6a44eb6b2145cd538b01af2 ({username='slave'} at 127.27.196.71:34057)
I20240502 21:29:31.797176  9533 tablet_copy_service.cc:161] P 19f162e164854f2da623a4bb8f9e6bf6: Beginning new tablet copy session on tablet 1497b781f36c43b39d22a05bb524c59d from peer 4cbb5b06a6a44eb6b2145cd538b01af2 at {username='slave'} at 127.27.196.71:34057: session id = 4cbb5b06a6a44eb6b2145cd538b01af2-1497b781f36c43b39d22a05bb524c59d
I20240502 21:29:31.800282  9533 tablet_copy_source_session.cc:205] T 1497b781f36c43b39d22a05bb524c59d P 19f162e164854f2da623a4bb8f9e6bf6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:31.802331 14877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1497b781f36c43b39d22a05bb524c59d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:31.804072  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:31.867764  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:29:31.868412  9378 consensus_queue.cc:237] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:31.876415  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:31.892750  9507 raft_consensus.cc:1261] T f0259f01acb249dd99c662151dc21916 P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:29:31.893972 14838 consensus_queue.cc:1035] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:31.916314 11365 raft_consensus.cc:1261] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:29:31.919471 14838 consensus_queue.cc:1035] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:31.933838 14877 tablet_copy_client.cc:783] T 1497b781f36c43b39d22a05bb524c59d P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:31.934425 14877 tablet_copy_client.cc:647] T 1497b781f36c43b39d22a05bb524c59d P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:31.941290 11364 raft_consensus.cc:2932] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:31.943346 14866 raft_consensus.cc:2932] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:31.962970  8963 catalog_manager.cc:5484] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: config changed from index 4 to 6, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New cstate: current_term: 2 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:31.971668 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:31.997787 14877 tablet_copy_client.cc:523] T 1497b781f36c43b39d22a05bb524c59d P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:32.015758  9085 tablet_service.cc:1514] Processing DeleteTablet for tablet f0259f01acb249dd99c662151dc21916 with delete_type TABLET_DATA_TOMBSTONED (TS 5543ddab5b484642b139b7c848932a58 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47814
I20240502 21:29:32.040398  9508 raft_consensus.cc:2932] T f0259f01acb249dd99c662151dc21916 P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:32.042328 14912 tablet_replica.cc:330] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58: stopping tablet replica
I20240502 21:29:32.042788 14912 raft_consensus.cc:2229] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:32.060478 14912 raft_consensus.cc:2258] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:32.064460 14912 ts_tablet_manager.cc:1875] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:32.067723  9356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:32.107386 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:32.113750 14877 tablet_bootstrap.cc:492] T 1497b781f36c43b39d22a05bb524c59d P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap starting.
I20240502 21:29:32.118805 14912 ts_tablet_manager.cc:1888] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 21:29:32.119066 14912 log.cc:1196] T f0259f01acb249dd99c662151dc21916 P 5543ddab5b484642b139b7c848932a58: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/wals/f0259f01acb249dd99c662151dc21916
I20240502 21:29:32.125687  8945 catalog_manager.cc:4830] TS 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): tablet f0259f01acb249dd99c662151dc21916 (table rebalance_test_table_4 [id=1b17408dbab9492bba0ed50efd6ef43f]) successfully deleted
I20240502 21:29:32.128024 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:32.393800 14877 tablet_bootstrap.cc:492] T 1497b781f36c43b39d22a05bb524c59d P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:32.394515 14877 tablet_bootstrap.cc:492] T 1497b781f36c43b39d22a05bb524c59d P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap complete.
I20240502 21:29:32.394950 14877 ts_tablet_manager.cc:1367] T 1497b781f36c43b39d22a05bb524c59d P 4cbb5b06a6a44eb6b2145cd538b01af2: Time spent bootstrapping tablet: real 0.281s	user 0.056s	sys 0.004s
I20240502 21:29:32.396255 14877 raft_consensus.cc:348] T 1497b781f36c43b39d22a05bb524c59d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:32.396809 14877 raft_consensus.cc:729] T 1497b781f36c43b39d22a05bb524c59d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbb5b06a6a44eb6b2145cd538b01af2, State: Initialized, Role: FOLLOWER
I20240502 21:29:32.397250 14877 consensus_queue.cc:260] T 1497b781f36c43b39d22a05bb524c59d P 4cbb5b06a6a44eb6b2145cd538b01af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:32.397994 14877 ts_tablet_manager.cc:1398] T 1497b781f36c43b39d22a05bb524c59d P 4cbb5b06a6a44eb6b2145cd538b01af2: Time spent starting tablet: real 0.003s	user 0.004s	sys 0.002s
I20240502 21:29:32.398929  9533 tablet_copy_service.cc:342] P 19f162e164854f2da623a4bb8f9e6bf6: Request end of tablet copy session 4cbb5b06a6a44eb6b2145cd538b01af2-1497b781f36c43b39d22a05bb524c59d received from {username='slave'} at 127.27.196.71:34057
I20240502 21:29:32.399091  9533 tablet_copy_service.cc:433] P 19f162e164854f2da623a4bb8f9e6bf6: ending tablet copy session 4cbb5b06a6a44eb6b2145cd538b01af2-1497b781f36c43b39d22a05bb524c59d on tablet 1497b781f36c43b39d22a05bb524c59d with peer 4cbb5b06a6a44eb6b2145cd538b01af2
I20240502 21:29:32.458195  9378 consensus_queue.cc:237] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:32.511925 11495 raft_consensus.cc:1261] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:32.512845 14866 consensus_queue.cc:1035] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:32.513828  9242 raft_consensus.cc:1261] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:32.504909  9241 consensus_queue.cc:237] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:29:32.517707 14866 consensus_queue.cc:1035] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:32.532089 11495 raft_consensus.cc:2932] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:32.530427 14866 raft_consensus.cc:2932] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:32.571167  9242 raft_consensus.cc:2932] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:32.584811  8956 catalog_manager.cc:5484] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 reported cstate change: config changed from index 4 to 5, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:32.592355  9507 raft_consensus.cc:1261] T 58c7004ceb044f5fa9dec54910046b69 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:32.592707  9378 raft_consensus.cc:1261] T 58c7004ceb044f5fa9dec54910046b69 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:32.593855 14837 consensus_queue.cc:1035] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:32.602808 14837 consensus_queue.cc:1035] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:32.629936 14617 raft_consensus.cc:2932] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } }
I20240502 21:29:32.633693  9378 raft_consensus.cc:2932] T 58c7004ceb044f5fa9dec54910046b69 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } }
I20240502 21:29:32.653923  9508 raft_consensus.cc:2932] T 58c7004ceb044f5fa9dec54910046b69 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } }
I20240502 21:29:32.684332  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:32.687945  8960 catalog_manager.cc:5484] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: config changed from index 4 to 5, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } }
I20240502 21:29:32.705535  9085 tablet_service.cc:1514] Processing DeleteTablet for tablet a7c6b881c9ef4cdca787f065778e526f with delete_type TABLET_DATA_TOMBSTONED (TS 5543ddab5b484642b139b7c848932a58 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47814
I20240502 21:29:32.721158 15022 tablet_replica.cc:330] T a7c6b881c9ef4cdca787f065778e526f P 5543ddab5b484642b139b7c848932a58: stopping tablet replica
I20240502 21:29:32.722543 15022 raft_consensus.cc:2229] T a7c6b881c9ef4cdca787f065778e526f P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:32.724648 15022 raft_consensus.cc:2258] T a7c6b881c9ef4cdca787f065778e526f P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:32.733127 15022 ts_tablet_manager.cc:1875] T a7c6b881c9ef4cdca787f065778e526f P 5543ddab5b484642b139b7c848932a58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:32.733856  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:29:32.740631 11209 tablet_service.cc:1514] Processing DeleteTablet for tablet 58c7004ceb044f5fa9dec54910046b69 with delete_type TABLET_DATA_TOMBSTONED (TS 6ef0beaf2f6941afb6ea9f9f274517fe not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36872
I20240502 21:29:32.742542 11364 raft_consensus.cc:1203] T 1497b781f36c43b39d22a05bb524c59d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 21:29:32.781009 15030 tablet_replica.cc:330] T 58c7004ceb044f5fa9dec54910046b69 P 6ef0beaf2f6941afb6ea9f9f274517fe: stopping tablet replica
I20240502 21:29:32.781798 15030 raft_consensus.cc:2229] T 58c7004ceb044f5fa9dec54910046b69 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:32.782315 15030 raft_consensus.cc:2258] T 58c7004ceb044f5fa9dec54910046b69 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:32.791529 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:32.802042 15022 ts_tablet_manager.cc:1888] T a7c6b881c9ef4cdca787f065778e526f P 5543ddab5b484642b139b7c848932a58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:29:32.802224 15022 log.cc:1196] T a7c6b881c9ef4cdca787f065778e526f P 5543ddab5b484642b139b7c848932a58: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/wals/a7c6b881c9ef4cdca787f065778e526f
I20240502 21:29:32.803440  8945 catalog_manager.cc:4830] TS 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): tablet a7c6b881c9ef4cdca787f065778e526f (table rebalance_test_table_6 [id=265d5656d7c048aa9795a906661b8ea8]) successfully deleted
I20240502 21:29:32.817315  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:32.818269 15030 ts_tablet_manager.cc:1875] T 58c7004ceb044f5fa9dec54910046b69 P 6ef0beaf2f6941afb6ea9f9f274517fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:32.820083  9356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:32.822268 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:32.840432 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:32.863142 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:32.934841 15030 ts_tablet_manager.cc:1888] T 58c7004ceb044f5fa9dec54910046b69 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:29:32.935321 15030 log.cc:1196] T 58c7004ceb044f5fa9dec54910046b69 P 6ef0beaf2f6941afb6ea9f9f274517fe: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal/wals/58c7004ceb044f5fa9dec54910046b69
I20240502 21:29:32.936854  8946 catalog_manager.cc:4830] TS 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): tablet 58c7004ceb044f5fa9dec54910046b69 (table rebalance_test_table_1 [id=f52c6c3804274777abfa00bb33ae9b16]) successfully deleted
I20240502 21:29:32.968662  9378 consensus_queue.cc:237] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:33.033821 11365 raft_consensus.cc:1261] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:33.034768 14866 consensus_queue.cc:1035] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:33.038228 11094 raft_consensus.cc:1261] T aeee59c5512d448d8f986f959dce1cc8 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:33.039178 14838 consensus_queue.cc:1035] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:33.042011  9105 raft_consensus.cc:1261] T aeee59c5512d448d8f986f959dce1cc8 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:33.045049 14838 consensus_queue.cc:1035] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:33.067698 14850 raft_consensus.cc:2932] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:33.074982 11365 raft_consensus.cc:2932] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:33.083490 11094 raft_consensus.cc:2932] T aeee59c5512d448d8f986f959dce1cc8 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:33.103101  8954 catalog_manager.cc:5484] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2 reported cstate change: config changed from index 5 to 6, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:33.114668  9490 tablet_service.cc:1514] Processing DeleteTablet for tablet aeee59c5512d448d8f986f959dce1cc8 with delete_type TABLET_DATA_TOMBSTONED (TS 19f162e164854f2da623a4bb8f9e6bf6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54670
I20240502 21:29:33.146327 15081 tablet_replica.cc:330] T aeee59c5512d448d8f986f959dce1cc8 P 19f162e164854f2da623a4bb8f9e6bf6: stopping tablet replica
I20240502 21:29:33.154503 15081 raft_consensus.cc:2229] T aeee59c5512d448d8f986f959dce1cc8 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:33.154976 15081 raft_consensus.cc:2258] T aeee59c5512d448d8f986f959dce1cc8 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:33.153846  9105 raft_consensus.cc:2932] T aeee59c5512d448d8f986f959dce1cc8 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:33.156066 15081 ts_tablet_manager.cc:1875] T aeee59c5512d448d8f986f959dce1cc8 P 19f162e164854f2da623a4bb8f9e6bf6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:33.189821  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:33.209321 15081 ts_tablet_manager.cc:1888] T aeee59c5512d448d8f986f959dce1cc8 P 19f162e164854f2da623a4bb8f9e6bf6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:29:33.236565 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:33.243280 15081 log.cc:1196] T aeee59c5512d448d8f986f959dce1cc8 P 19f162e164854f2da623a4bb8f9e6bf6: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/wals/aeee59c5512d448d8f986f959dce1cc8
I20240502 21:29:33.247355  8945 catalog_manager.cc:4830] TS 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): tablet aeee59c5512d448d8f986f959dce1cc8 (table rebalance_test_table_2 [id=c0a0ef39a2a246449eb582cc1d993484]) successfully deleted
I20240502 21:29:33.290576  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:33.320927 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:33.347381  9356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:33.350801 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:33.376479 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:33.380551  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:29:33.604599  9378 consensus_queue.cc:237] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:33.608438 11365 raft_consensus.cc:1261] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:33.610622 14850 consensus_queue.cc:1035] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:33.615182 11094 raft_consensus.cc:1261] T aeee59c5512d448d8f986f959dce1cc8 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:33.614909 14850 raft_consensus.cc:2932] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:33.621603 11365 raft_consensus.cc:2932] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:33.629865 14866 consensus_queue.cc:1035] T aeee59c5512d448d8f986f959dce1cc8 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.012s
W20240502 21:29:33.691745  9310 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.65:42177, user_credentials={real_user=slave}} blocked reactor thread for 71248.6us
I20240502 21:29:33.700877  8960 catalog_manager.cc:5484] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2 reported cstate change: config changed from index 6 to 7, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
W20240502 21:29:33.706466  9311 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.69:42049, user_credentials={real_user=slave}} blocked reactor thread for 89086.8us
I20240502 21:29:33.710254 11094 raft_consensus.cc:2932] T aeee59c5512d448d8f986f959dce1cc8 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:33.729055  9085 tablet_service.cc:1514] Processing DeleteTablet for tablet aeee59c5512d448d8f986f959dce1cc8 with delete_type TABLET_DATA_TOMBSTONED (TS 5543ddab5b484642b139b7c848932a58 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47814
I20240502 21:29:33.729959 15166 tablet_replica.cc:330] T aeee59c5512d448d8f986f959dce1cc8 P 5543ddab5b484642b139b7c848932a58: stopping tablet replica
I20240502 21:29:33.730216 15166 raft_consensus.cc:2229] T aeee59c5512d448d8f986f959dce1cc8 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:33.730696 15166 raft_consensus.cc:2258] T aeee59c5512d448d8f986f959dce1cc8 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:33.732318 15166 ts_tablet_manager.cc:1875] T aeee59c5512d448d8f986f959dce1cc8 P 5543ddab5b484642b139b7c848932a58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:33.746770 15166 ts_tablet_manager.cc:1888] T aeee59c5512d448d8f986f959dce1cc8 P 5543ddab5b484642b139b7c848932a58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:29:33.749481 15166 log.cc:1196] T aeee59c5512d448d8f986f959dce1cc8 P 5543ddab5b484642b139b7c848932a58: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/wals/aeee59c5512d448d8f986f959dce1cc8
I20240502 21:29:33.750859  8945 catalog_manager.cc:4830] TS 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): tablet aeee59c5512d448d8f986f959dce1cc8 (table rebalance_test_table_2 [id=c0a0ef39a2a246449eb582cc1d993484]) successfully deleted
W20240502 21:29:33.951301  9444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.65:42177, user_credentials={real_user=slave}} blocked reactor thread for 66526.7us
I20240502 21:29:33.964444  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:33.978775  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:33.980569 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:34.001844  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:34.028206  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:34.063705 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:34.123265  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:29:34.153419 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:34.178753 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:34.205757  9356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:34.236873  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:29:34.239197 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:34.270502 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:34.294492 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:34.325625 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:34.372707  9356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:34.723289  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:34.781364  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:34.799492 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:34.915527 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:34.945142 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:34.994115  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:29:35.002620 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:35.028220  9356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:35.179658  9105 consensus_queue.cc:237] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:35.197742 11092 raft_consensus.cc:1261] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:35.198642 13470 consensus_queue.cc:1035] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:35.199342 11496 raft_consensus.cc:1261] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:35.201906  9378 raft_consensus.cc:1261] T 79dedaa327b6479e8325f91481c1688e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:35.202711 13470 consensus_queue.cc:1035] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:35.210410 13470 consensus_queue.cc:1035] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:35.256242  9377 consensus_queue.cc:237] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:35.285661 13470 raft_consensus.cc:2932] T 79dedaa327b6479e8325f91481c1688e P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:35.291277 11495 raft_consensus.cc:2932] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:35.307900 11092 raft_consensus.cc:2932] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:35.322880 11094 raft_consensus.cc:1261] T 8a2e06253c51405fa9b6bc9e87844760 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:35.324121 15161 consensus_queue.cc:1035] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:35.324419  9508 raft_consensus.cc:1261] T 8a2e06253c51405fa9b6bc9e87844760 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:35.326359  8953 catalog_manager.cc:5484] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2 reported cstate change: config changed from index 5 to 6, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:35.328562 15248 consensus_queue.cc:1035] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:35.378399 15161 raft_consensus.cc:2932] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:35.407919 11094 raft_consensus.cc:2932] T 8a2e06253c51405fa9b6bc9e87844760 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:35.443095  8956 catalog_manager.cc:5484] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: config changed from index 4 to 5, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:35.443172  9508 raft_consensus.cc:2932] T 8a2e06253c51405fa9b6bc9e87844760 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
W20240502 21:29:35.448511  9040 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.67:43203, user_credentials={real_user=slave}} blocked reactor thread for 76396.5us
I20240502 21:29:35.449774  9378 raft_consensus.cc:2932] T 79dedaa327b6479e8325f91481c1688e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:35.477039  9490 tablet_service.cc:1514] Processing DeleteTablet for tablet 79dedaa327b6479e8325f91481c1688e with delete_type TABLET_DATA_TOMBSTONED (TS 19f162e164854f2da623a4bb8f9e6bf6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54670
I20240502 21:29:35.491030 15384 tablet_replica.cc:330] T 79dedaa327b6479e8325f91481c1688e P 19f162e164854f2da623a4bb8f9e6bf6: stopping tablet replica
I20240502 21:29:35.491556 15384 raft_consensus.cc:2229] T 79dedaa327b6479e8325f91481c1688e P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:35.492058 15384 raft_consensus.cc:2258] T 79dedaa327b6479e8325f91481c1688e P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:35.493348 15384 ts_tablet_manager.cc:1875] T 79dedaa327b6479e8325f91481c1688e P 19f162e164854f2da623a4bb8f9e6bf6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:35.506315  9221 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a2e06253c51405fa9b6bc9e87844760 with delete_type TABLET_DATA_TOMBSTONED (TS 525a093ea5bf40dc9bbf9dc39895d670 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44408
I20240502 21:29:35.544912 15384 ts_tablet_manager.cc:1888] T 79dedaa327b6479e8325f91481c1688e P 19f162e164854f2da623a4bb8f9e6bf6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:29:35.546026 15389 tablet_replica.cc:330] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670: stopping tablet replica
I20240502 21:29:35.546594 15389 raft_consensus.cc:2229] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:35.547086 15389 raft_consensus.cc:2258] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:35.546943 15384 log.cc:1196] T 79dedaa327b6479e8325f91481c1688e P 19f162e164854f2da623a4bb8f9e6bf6: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/wals/79dedaa327b6479e8325f91481c1688e
I20240502 21:29:35.548310 15389 ts_tablet_manager.cc:1875] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:35.549880  8945 catalog_manager.cc:4830] TS 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): tablet 79dedaa327b6479e8325f91481c1688e (table rebalance_test_table_7 [id=ef412fa8fe9249729cb99879a5c8343a]) successfully deleted
I20240502 21:29:35.551148  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:35.558943 15389 ts_tablet_manager.cc:1888] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:29:35.559556 15389 log.cc:1196] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/wals/8a2e06253c51405fa9b6bc9e87844760
I20240502 21:29:35.561437  8949 catalog_manager.cc:4830] TS 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): tablet 8a2e06253c51405fa9b6bc9e87844760 (table rebalance_test_table_5 [id=e8eef05f850243d0bdc7e1899f8c6139]) successfully deleted
I20240502 21:29:35.599360  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:35.650211 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:35.651286 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:35.666133 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:35.671629  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:29:35.697767 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:35.706471  9356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:35.876830  9107 consensus_queue.cc:237] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:35.890112 11094 raft_consensus.cc:1261] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:35.891978  9508 raft_consensus.cc:1261] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:35.895440 15366 consensus_queue.cc:1035] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:35.896250 15366 consensus_queue.cc:1035] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:35.899461 11364 raft_consensus.cc:1261] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:35.902465 15363 consensus_queue.cc:1035] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:35.917569 15366 raft_consensus.cc:2932] T 8b4486f03fd9474aa0bb2cb4c9149437 P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:35.922132  9507 raft_consensus.cc:2932] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:35.955864  8962 catalog_manager.cc:5484] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: config changed from index 5 to 6, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:35.963963  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:35.973129  9356 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b4486f03fd9474aa0bb2cb4c9149437 with delete_type TABLET_DATA_TOMBSTONED (TS f2fdd596a8b44bbc848fe40f9f07fbe0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55106
I20240502 21:29:35.989368 15473 tablet_replica.cc:330] T 8b4486f03fd9474aa0bb2cb4c9149437 P f2fdd596a8b44bbc848fe40f9f07fbe0: stopping tablet replica
I20240502 21:29:35.989802 15473 raft_consensus.cc:2229] T 8b4486f03fd9474aa0bb2cb4c9149437 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:35.990257 15473 raft_consensus.cc:2258] T 8b4486f03fd9474aa0bb2cb4c9149437 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:35.995406  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:36.005955 11364 raft_consensus.cc:2932] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:36.008111 11073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:36.043466 11094 raft_consensus.cc:2932] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:36.056223 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:36.072273 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:36.077692 15473 ts_tablet_manager.cc:1875] T 8b4486f03fd9474aa0bb2cb4c9149437 P f2fdd596a8b44bbc848fe40f9f07fbe0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:36.078078  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:29:36.155128 15473 ts_tablet_manager.cc:1888] T 8b4486f03fd9474aa0bb2cb4c9149437 P f2fdd596a8b44bbc848fe40f9f07fbe0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:29:36.160849 15473 log.cc:1196] T 8b4486f03fd9474aa0bb2cb4c9149437 P f2fdd596a8b44bbc848fe40f9f07fbe0: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/wals/8b4486f03fd9474aa0bb2cb4c9149437
I20240502 21:29:36.170861  8948 catalog_manager.cc:4830] TS f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): tablet 8b4486f03fd9474aa0bb2cb4c9149437 (table rebalance_test_table_3 [id=a237d7bda970485480e98020c6ac738f]) successfully deleted
I20240502 21:29:36.193012 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:36.215181  9378 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f9ae8fac693d4d68b6741f2509e1265e"
dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:50798
I20240502 21:29:36.215529  9378 raft_consensus.cc:595] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:29:36.225260  9356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:36.359553 15161 raft_consensus.cc:979] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0: : Instructing follower d10e9e156ba74a97a221c85b68ab8bc4 to start an election
I20240502 21:29:36.359772 15161 raft_consensus.cc:1067] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Signalling peer d10e9e156ba74a97a221c85b68ab8bc4 to start an election
I20240502 21:29:36.361078 11094 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f9ae8fac693d4d68b6741f2509e1265e"
dest_uuid: "d10e9e156ba74a97a221c85b68ab8bc4"
 from {username='slave'} at 127.27.196.67:54351
I20240502 21:29:36.361512 11094 raft_consensus.cc:482] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:36.361765 11094 raft_consensus.cc:3037] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:36.388420 11094 raft_consensus.cc:504] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:36.393069  9107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ae8fac693d4d68b6741f2509e1265e" candidate_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "5543ddab5b484642b139b7c848932a58"
I20240502 21:29:36.393453  9107 raft_consensus.cc:3037] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:36.404560  9107 raft_consensus.cc:2454] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d10e9e156ba74a97a221c85b68ab8bc4 in term 2.
I20240502 21:29:36.410316  9242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ae8fac693d4d68b6741f2509e1265e" candidate_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670"
I20240502 21:29:36.410702  9242 raft_consensus.cc:3037] T f9ae8fac693d4d68b6741f2509e1265e P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:36.416321  9242 raft_consensus.cc:2454] T f9ae8fac693d4d68b6741f2509e1265e P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d10e9e156ba74a97a221c85b68ab8bc4 in term 2.
I20240502 21:29:36.417084 11028 leader_election.cc:302] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 525a093ea5bf40dc9bbf9dc39895d670, 5543ddab5b484642b139b7c848932a58, d10e9e156ba74a97a221c85b68ab8bc4; no voters: 
I20240502 21:29:36.418215 15482 raft_consensus.cc:2781] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:36.422205  9378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ae8fac693d4d68b6741f2509e1265e" candidate_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
I20240502 21:29:36.424803  9378 raft_consensus.cc:3032] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:36.424944  9378 raft_consensus.cc:729] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Running, Role: LEADER
I20240502 21:29:36.425372  9378 consensus_queue.cc:260] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:36.425889  9378 raft_consensus.cc:3037] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:36.426170  9107 raft_consensus.cc:1226] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Rejecting Update request from peer f2fdd596a8b44bbc848fe40f9f07fbe0 for earlier term 1. Current term is 2. Ops: []
I20240502 21:29:36.430059  9378 raft_consensus.cc:2454] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d10e9e156ba74a97a221c85b68ab8bc4 in term 2.
I20240502 21:29:36.445840 11094 leader_election.cc:288] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 election: Requested vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467)
I20240502 21:29:36.461764 15482 raft_consensus.cc:686] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 LEADER]: Becoming Leader. State: Replica: d10e9e156ba74a97a221c85b68ab8bc4, State: Running, Role: LEADER
I20240502 21:29:36.462595 15482 consensus_queue.cc:237] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:36.474963 11496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ae8fac693d4d68b6741f2509e1265e" candidate_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e0de0545fa0547d3929111e990e6d1b2"
I20240502 21:29:36.475495 11496 raft_consensus.cc:3037] T f9ae8fac693d4d68b6741f2509e1265e P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:36.481590 11496 raft_consensus.cc:2454] T f9ae8fac693d4d68b6741f2509e1265e P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d10e9e156ba74a97a221c85b68ab8bc4 in term 2.
I20240502 21:29:36.484050  8962 catalog_manager.cc:5484] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 reported cstate change: term changed from 1 to 2, leader changed from f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) to d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69). New cstate: current_term: 2 leader_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:36.687367  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:36.721580  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:36.743582 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:36.757473  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:29:36.782485 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:36.805771 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:36.823000 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:36.859434  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:36.891058  9107 raft_consensus.cc:1261] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Refusing update from remote peer d10e9e156ba74a97a221c85b68ab8bc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:29:36.892282 15482 consensus_queue.cc:1035] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:36.910279 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:36.938977  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:36.940420  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:36.971488  9378 raft_consensus.cc:1261] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Refusing update from remote peer d10e9e156ba74a97a221c85b68ab8bc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:29:36.972278 15482 consensus_queue.cc:1035] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:36.998191  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:29:37.035039 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:37.081049 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:37.096541 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:37.113006 11496 raft_consensus.cc:1261] T f9ae8fac693d4d68b6741f2509e1265e P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Refusing update from remote peer d10e9e156ba74a97a221c85b68ab8bc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:29:37.113898  9242 raft_consensus.cc:1261] T f9ae8fac693d4d68b6741f2509e1265e P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Refusing update from remote peer d10e9e156ba74a97a221c85b68ab8bc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:29:37.126649 15624 consensus_queue.cc:1035] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:37.131258 15482 consensus_queue.cc:1035] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:37.147101  9377 consensus_queue.cc:237] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:37.155442 11229 raft_consensus.cc:1261] T 4674e376bf7440cda619784491685b44 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:37.156810 15518 consensus_queue.cc:1035] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:37.170552  9105 raft_consensus.cc:1261] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:37.172044 15518 consensus_queue.cc:1035] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:37.218479 15515 raft_consensus.cc:2932] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:37.220681  9107 raft_consensus.cc:2932] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:37.253891  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:37.270062  8963 catalog_manager.cc:5484] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: config changed from index 4 to 5, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:37.285043 11228 raft_consensus.cc:2932] T 4674e376bf7440cda619784491685b44 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:37.334233  9221 tablet_service.cc:1514] Processing DeleteTablet for tablet 4674e376bf7440cda619784491685b44 with delete_type TABLET_DATA_TOMBSTONED (TS 525a093ea5bf40dc9bbf9dc39895d670 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44408
I20240502 21:29:37.350775 15669 tablet_replica.cc:330] T 4674e376bf7440cda619784491685b44 P 525a093ea5bf40dc9bbf9dc39895d670: stopping tablet replica
I20240502 21:29:37.353065 15669 raft_consensus.cc:2229] T 4674e376bf7440cda619784491685b44 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:37.353482 15669 raft_consensus.cc:2258] T 4674e376bf7440cda619784491685b44 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:37.354486 15669 ts_tablet_manager.cc:1875] T 4674e376bf7440cda619784491685b44 P 525a093ea5bf40dc9bbf9dc39895d670: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:37.380156 15669 ts_tablet_manager.cc:1888] T 4674e376bf7440cda619784491685b44 P 525a093ea5bf40dc9bbf9dc39895d670: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:29:37.387351 15669 log.cc:1196] T 4674e376bf7440cda619784491685b44 P 525a093ea5bf40dc9bbf9dc39895d670: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/wals/4674e376bf7440cda619784491685b44
I20240502 21:29:37.389433  8949 catalog_manager.cc:4830] TS 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): tablet 4674e376bf7440cda619784491685b44 (table rebalance_test_table_3 [id=a237d7bda970485480e98020c6ac738f]) successfully deleted
I20240502 21:29:37.509583  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:37.513687 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:37.553728  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:29:37.616621 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:37.665901  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:37.666764  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:37.679471 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:37.685424 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:37.686618  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:37.695892 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:37.720865  9221 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:37.730970 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:37.744472 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:37.749183 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:37.755885  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:37.793020  9105 consensus_queue.cc:237] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:37.824227 11092 raft_consensus.cc:1261] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:37.825107 15363 consensus_queue.cc:1035] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:37.828557  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:29:37.832777  9242 raft_consensus.cc:1261] T e46065d94e874293a836bc74241b8e06 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:37.833565 15363 consensus_queue.cc:1035] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:37.908859 15363 raft_consensus.cc:2932] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:37.916162  9242 raft_consensus.cc:2932] T e46065d94e874293a836bc74241b8e06 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:37.918009  8962 catalog_manager.cc:5484] T e46065d94e874293a836bc74241b8e06 P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index 4 to 5, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:37.936978 11092 raft_consensus.cc:2932] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:37.965093  9356 tablet_service.cc:1514] Processing DeleteTablet for tablet e46065d94e874293a836bc74241b8e06 with delete_type TABLET_DATA_TOMBSTONED (TS f2fdd596a8b44bbc848fe40f9f07fbe0 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55106
I20240502 21:29:37.991668 15747 tablet_replica.cc:330] T e46065d94e874293a836bc74241b8e06 P f2fdd596a8b44bbc848fe40f9f07fbe0: stopping tablet replica
I20240502 21:29:37.992105 15747 raft_consensus.cc:2229] T e46065d94e874293a836bc74241b8e06 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:38.012595 15747 raft_consensus.cc:2258] T e46065d94e874293a836bc74241b8e06 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:38.025182 15747 ts_tablet_manager.cc:1875] T e46065d94e874293a836bc74241b8e06 P f2fdd596a8b44bbc848fe40f9f07fbe0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:38.080159 15747 ts_tablet_manager.cc:1888] T e46065d94e874293a836bc74241b8e06 P f2fdd596a8b44bbc848fe40f9f07fbe0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:29:38.080307 15747 log.cc:1196] T e46065d94e874293a836bc74241b8e06 P f2fdd596a8b44bbc848fe40f9f07fbe0: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/wals/e46065d94e874293a836bc74241b8e06
I20240502 21:29:38.084519  8948 catalog_manager.cc:4830] TS f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): tablet e46065d94e874293a836bc74241b8e06 (table rebalance_test_table_8 [id=5802da0404e74a07bca290cdc59b279b]) successfully deleted
I20240502 21:29:38.114342  9507 consensus_queue.cc:237] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:38.133796 11228 raft_consensus.cc:1261] T c56ac926025e47218e9df4088184a278 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:38.136119 15617 consensus_queue.cc:1035] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:38.138509  9242 raft_consensus.cc:1261] T c56ac926025e47218e9df4088184a278 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:38.139483 15617 consensus_queue.cc:1035] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:38.155985 15617 raft_consensus.cc:2932] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:38.171666 11228 raft_consensus.cc:2932] T c56ac926025e47218e9df4088184a278 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:38.202898  8961 catalog_manager.cc:5484] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: config changed from index 4 to 5, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New cstate: current_term: 1 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:38.209957  9085 tablet_service.cc:1514] Processing DeleteTablet for tablet c56ac926025e47218e9df4088184a278 with delete_type TABLET_DATA_TOMBSTONED (TS 5543ddab5b484642b139b7c848932a58 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47814
I20240502 21:29:38.241379  9239 raft_consensus.cc:2932] T c56ac926025e47218e9df4088184a278 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:38.253381 15794 tablet_replica.cc:330] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58: stopping tablet replica
I20240502 21:29:38.253793 15794 raft_consensus.cc:2229] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:38.254112 15794 raft_consensus.cc:2258] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:38.259488 15794 ts_tablet_manager.cc:1875] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:38.297470 15794 ts_tablet_manager.cc:1888] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:29:38.297623 15794 log.cc:1196] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/wals/c56ac926025e47218e9df4088184a278
I20240502 21:29:38.304656  8945 catalog_manager.cc:4830] TS 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): tablet c56ac926025e47218e9df4088184a278 (table rebalance_test_table_8 [id=5802da0404e74a07bca290cdc59b279b]) successfully deleted
I20240502 21:29:38.363714 11092 consensus_queue.cc:237] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:38.374358  9239 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "07d693b680a54e12871121737e3da744"
dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:52570
I20240502 21:29:38.374709  9239 raft_consensus.cc:595] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:29:38.377301 11496 raft_consensus.cc:1261] T f9ae8fac693d4d68b6741f2509e1265e P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Refusing update from remote peer d10e9e156ba74a97a221c85b68ab8bc4: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:29:38.378193 15774 consensus_queue.cc:1035] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:38.385838  9239 raft_consensus.cc:1261] T f9ae8fac693d4d68b6741f2509e1265e P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Refusing update from remote peer d10e9e156ba74a97a221c85b68ab8bc4: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:29:38.386731 15774 consensus_queue.cc:1035] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:38.397886  9105 raft_consensus.cc:1261] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Refusing update from remote peer d10e9e156ba74a97a221c85b68ab8bc4: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:29:38.398808 15774 consensus_queue.cc:1035] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:38.423610 15774 raft_consensus.cc:2932] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:38.427848  9239 raft_consensus.cc:2932] T f9ae8fac693d4d68b6741f2509e1265e P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:38.431362 15027 raft_consensus.cc:979] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670: : Instructing follower 6ef0beaf2f6941afb6ea9f9f274517fe to start an election
I20240502 21:29:38.431656 15027 raft_consensus.cc:1067] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Signalling peer 6ef0beaf2f6941afb6ea9f9f274517fe to start an election
I20240502 21:29:38.433472 11228 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "07d693b680a54e12871121737e3da744"
dest_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe"
 from {username='slave'} at 127.27.196.66:33219
I20240502 21:29:38.433981 11228 raft_consensus.cc:482] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:38.434273 11228 raft_consensus.cc:3037] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:38.440677  8962 catalog_manager.cc:5484] T f9ae8fac693d4d68b6741f2509e1265e P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: config changed from index 5 to 7, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New cstate: current_term: 2 leader_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:38.444003 11228 raft_consensus.cc:504] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:38.447528  9105 raft_consensus.cc:2932] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:38.448691  9239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d693b680a54e12871121737e3da744" candidate_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670"
I20240502 21:29:38.449031  9239 raft_consensus.cc:3032] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:38.449216  9239 raft_consensus.cc:729] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Running, Role: LEADER
I20240502 21:29:38.449890  9239 consensus_queue.cc:260] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:38.450151  9508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d693b680a54e12871121737e3da744" candidate_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6"
I20240502 21:29:38.450665  9508 raft_consensus.cc:3037] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:38.450987  9239 raft_consensus.cc:3037] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:38.453519  9356 tablet_service.cc:1514] Processing DeleteTablet for tablet f9ae8fac693d4d68b6741f2509e1265e with delete_type TABLET_DATA_TOMBSTONED (TS f2fdd596a8b44bbc848fe40f9f07fbe0 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55106
I20240502 21:29:38.454120 15747 tablet_replica.cc:330] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0: stopping tablet replica
I20240502 21:29:38.454634 15747 raft_consensus.cc:2229] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:38.454968 15747 raft_consensus.cc:2258] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:38.456041 15747 ts_tablet_manager.cc:1875] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:38.457224  9239 raft_consensus.cc:2454] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef0beaf2f6941afb6ea9f9f274517fe in term 2.
I20240502 21:29:38.460954  9378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d693b680a54e12871121737e3da744" candidate_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
I20240502 21:29:38.461256  9378 raft_consensus.cc:3037] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:38.464872  9378 raft_consensus.cc:2454] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef0beaf2f6941afb6ea9f9f274517fe in term 2.
I20240502 21:29:38.465979 15747 ts_tablet_manager.cc:1888] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 21:29:38.466228 15747 log.cc:1196] T f9ae8fac693d4d68b6741f2509e1265e P f2fdd596a8b44bbc848fe40f9f07fbe0: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/wals/f9ae8fac693d4d68b6741f2509e1265e
I20240502 21:29:38.467330  8948 catalog_manager.cc:4830] TS f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): tablet f9ae8fac693d4d68b6741f2509e1265e (table rebalance_test_table_6 [id=265d5656d7c048aa9795a906661b8ea8]) successfully deleted
I20240502 21:29:38.484246 11228 leader_election.cc:288] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [CANDIDATE]: Term 2 election: Requested vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049), 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857)
I20240502 21:29:38.495317 11495 raft_consensus.cc:2932] T f9ae8fac693d4d68b6741f2509e1265e P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:38.503396  9508 raft_consensus.cc:2454] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef0beaf2f6941afb6ea9f9f274517fe in term 2.
I20240502 21:29:38.504490 11160 leader_election.cc:302] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 6 voters: 4 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 525a093ea5bf40dc9bbf9dc39895d670, 6ef0beaf2f6941afb6ea9f9f274517fe, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:29:38.505079 15790 raft_consensus.cc:2781] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:38.519721 15790 raft_consensus.cc:686] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 LEADER]: Becoming Leader. State: Replica: 6ef0beaf2f6941afb6ea9f9f274517fe, State: Running, Role: LEADER
I20240502 21:29:38.520538 15790 consensus_queue.cc:237] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:38.539045 11364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d693b680a54e12871121737e3da744" candidate_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2"
I20240502 21:29:38.539387 11364 raft_consensus.cc:3037] T 07d693b680a54e12871121737e3da744 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:38.542714 11364 raft_consensus.cc:2454] T 07d693b680a54e12871121737e3da744 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef0beaf2f6941afb6ea9f9f274517fe in term 2.
I20240502 21:29:38.568161 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:38.583475  9221 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:38.625059 11094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d693b680a54e12871121737e3da744" candidate_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "d10e9e156ba74a97a221c85b68ab8bc4"
I20240502 21:29:38.625506 11094 raft_consensus.cc:3037] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:38.629822 11094 raft_consensus.cc:2454] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef0beaf2f6941afb6ea9f9f274517fe in term 2.
I20240502 21:29:38.638175  8955 catalog_manager.cc:5484] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe reported cstate change: term changed from 1 to 2, leader changed from 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) to 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70). New cstate: current_term: 2 leader_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:38.653146  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:29:38.660634  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:38.661842  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:38.706223 11209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:38.720417 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:38.748308 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:38.924863 11364 raft_consensus.cc:1261] T 07d693b680a54e12871121737e3da744 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Refusing update from remote peer 6ef0beaf2f6941afb6ea9f9f274517fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:29:38.925987  9508 raft_consensus.cc:1261] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Refusing update from remote peer 6ef0beaf2f6941afb6ea9f9f274517fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:29:38.928339 15844 consensus_queue.cc:1035] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:38.931474 15844 consensus_queue.cc:1035] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:39.003412  9378 raft_consensus.cc:1261] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Refusing update from remote peer 6ef0beaf2f6941afb6ea9f9f274517fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:29:39.004556 15844 consensus_queue.cc:1035] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:39.033933  9242 raft_consensus.cc:1261] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Refusing update from remote peer 6ef0beaf2f6941afb6ea9f9f274517fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:29:39.035646 15844 consensus_queue.cc:1035] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240502 21:29:39.051710 11092 raft_consensus.cc:1261] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Refusing update from remote peer 6ef0beaf2f6941afb6ea9f9f274517fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:29:39.052973 15844 consensus_queue.cc:1035] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:39.235916  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:39.242718 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:39.327567  9221 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:39.338601 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:29:39.381594  9311 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.69:42049, user_credentials={real_user=slave}} blocked reactor thread for 86878us
I20240502 21:29:39.391644 11209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:39.421922  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:29:39.432008  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:39.439877 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:39.673588 11209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:39.690320 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:39.698755  9221 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:39.705686  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:29:39.708114 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:39.714537  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:39.780061  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:39.827788 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:39.861007  9221 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:39.865578  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:39.877352 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:39.890321 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:39.919462  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:29:39.919602 11209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:39.982364  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:40.008965  9242 consensus_queue.cc:237] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:29:40.033735 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:40.093974  9105 raft_consensus.cc:1261] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:40.094897 15027 consensus_queue.cc:1035] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:40.127878 15027 raft_consensus.cc:2932] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } }
I20240502 21:29:40.142843  9105 raft_consensus.cc:2932] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } }
I20240502 21:29:40.162314  8958 catalog_manager.cc:5484] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index 5 to 6, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } }
I20240502 21:29:40.199924  9221 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 21:29:40.222151 11162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.67:43203, user_credentials={real_user=slave}} blocked reactor thread for 73762.9us
W20240502 21:29:40.222693 11160 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.68:40931, user_credentials={real_user=slave}} blocked reactor thread for 71909.2us
I20240502 21:29:40.281670 11209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:40.299441  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:40.308607 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:40.326112 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:40.355938 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:40.400271  9356 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d1a31513bb64578a85adf1f622e5bc0 with delete_type TABLET_DATA_TOMBSTONED (TS f2fdd596a8b44bbc848fe40f9f07fbe0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55106
I20240502 21:29:40.403280  9242 consensus_queue.cc:237] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
W20240502 21:29:40.418243  9173 consensus_peers.cc:485] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:40.425123  9105 raft_consensus.cc:1261] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:40.426079 15027 consensus_queue.cc:1035] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:40.433562 15027 raft_consensus.cc:2932] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:40.438144  9107 raft_consensus.cc:2932] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:40.449052  8962 catalog_manager.cc:5484] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index 6 to 7, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:40.451983  8949 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 4d1a31513bb64578a85adf1f622e5bc0 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 21:29:40.527001  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:29:40.546290 16074 tablet_replica.cc:330] T 4d1a31513bb64578a85adf1f622e5bc0 P f2fdd596a8b44bbc848fe40f9f07fbe0: stopping tablet replica
I20240502 21:29:40.555732  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:40.571291 16074 raft_consensus.cc:2229] T 4d1a31513bb64578a85adf1f622e5bc0 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:40.572080 16074 raft_consensus.cc:2258] T 4d1a31513bb64578a85adf1f622e5bc0 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:40.573062 16074 ts_tablet_manager.cc:1875] T 4d1a31513bb64578a85adf1f622e5bc0 P f2fdd596a8b44bbc848fe40f9f07fbe0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:40.583887 16074 ts_tablet_manager.cc:1888] T 4d1a31513bb64578a85adf1f622e5bc0 P f2fdd596a8b44bbc848fe40f9f07fbe0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:29:40.584108 16074 log.cc:1196] T 4d1a31513bb64578a85adf1f622e5bc0 P f2fdd596a8b44bbc848fe40f9f07fbe0: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/wals/4d1a31513bb64578a85adf1f622e5bc0
I20240502 21:29:40.585067  8948 catalog_manager.cc:4830] TS f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): tablet 4d1a31513bb64578a85adf1f622e5bc0 (table rebalance_test_table_1 [id=f52c6c3804274777abfa00bb33ae9b16]) successfully deleted
I20240502 21:29:40.657634 11092 consensus_queue.cc:237] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:40.687844 11495 raft_consensus.cc:1261] T f9ae8fac693d4d68b6741f2509e1265e P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Refusing update from remote peer d10e9e156ba74a97a221c85b68ab8bc4: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 21:29:40.693131 15774 consensus_queue.cc:1035] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20240502 21:29:40.696621  9242 raft_consensus.cc:1261] T f9ae8fac693d4d68b6741f2509e1265e P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Refusing update from remote peer d10e9e156ba74a97a221c85b68ab8bc4: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 21:29:40.697889 15774 consensus_queue.cc:1035] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:40.743552 16111 raft_consensus.cc:2932] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:40.753355  8962 catalog_manager.cc:5484] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 reported cstate change: config changed from index 7 to 8, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New cstate: current_term: 2 leader_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:40.755731  9242 raft_consensus.cc:2932] T f9ae8fac693d4d68b6741f2509e1265e P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:40.750097 11495 raft_consensus.cc:2932] T f9ae8fac693d4d68b6741f2509e1265e P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:40.759258  9085 tablet_service.cc:1514] Processing DeleteTablet for tablet f9ae8fac693d4d68b6741f2509e1265e with delete_type TABLET_DATA_TOMBSTONED (TS 5543ddab5b484642b139b7c848932a58 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:47814
I20240502 21:29:40.810822 16118 tablet_replica.cc:330] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58: stopping tablet replica
I20240502 21:29:40.811789 16118 raft_consensus.cc:2229] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:40.812279 16118 raft_consensus.cc:2258] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:40.813460 16118 ts_tablet_manager.cc:1875] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:40.838944 16131 ts_tablet_manager.cc:902] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: Initiating tablet copy from peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:29:40.847218 16118 ts_tablet_manager.cc:1888] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 21:29:40.847985 16118 log.cc:1196] T f9ae8fac693d4d68b6741f2509e1265e P 5543ddab5b484642b139b7c848932a58: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/wals/f9ae8fac693d4d68b6741f2509e1265e
I20240502 21:29:40.850442  8945 catalog_manager.cc:4830] TS 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): tablet f9ae8fac693d4d68b6741f2509e1265e (table rebalance_test_table_6 [id=265d5656d7c048aa9795a906661b8ea8]) successfully deleted
I20240502 21:29:40.934975 16131 tablet_copy_client.cc:308] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:39071
I20240502 21:29:40.937013  9263 tablet_copy_service.cc:140] P 525a093ea5bf40dc9bbf9dc39895d670: Received BeginTabletCopySession request for tablet 4d1a31513bb64578a85adf1f622e5bc0 from peer 6ef0beaf2f6941afb6ea9f9f274517fe ({username='slave'} at 127.27.196.70:52575)
I20240502 21:29:40.951471  9263 tablet_copy_service.cc:161] P 525a093ea5bf40dc9bbf9dc39895d670: Beginning new tablet copy session on tablet 4d1a31513bb64578a85adf1f622e5bc0 from peer 6ef0beaf2f6941afb6ea9f9f274517fe at {username='slave'} at 127.27.196.70:52575: session id = 6ef0beaf2f6941afb6ea9f9f274517fe-4d1a31513bb64578a85adf1f622e5bc0
I20240502 21:29:40.958685  9263 tablet_copy_source_session.cc:205] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:40.961591 16131 ts_tablet_manager.cc:1875] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:29:40.980427 16131 ts_tablet_manager.cc:1888] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 21:29:40.982401 16131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d1a31513bb64578a85adf1f622e5bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:40.988502 16131 tablet_copy_client.cc:783] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:40.988865 16131 tablet_copy_client.cc:647] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:40.991799 16131 tablet_copy_client.cc:523] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:40.998833 16131 tablet_bootstrap.cc:492] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap starting.
I20240502 21:29:41.129510 16131 tablet_bootstrap.cc:492] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:41.130657 16131 tablet_bootstrap.cc:492] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap complete.
I20240502 21:29:41.131187 16131 ts_tablet_manager.cc:1367] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent bootstrapping tablet: real 0.133s	user 0.064s	sys 0.018s
I20240502 21:29:41.132220 16131 raft_consensus.cc:348] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:41.132522 16131 raft_consensus.cc:729] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef0beaf2f6941afb6ea9f9f274517fe, State: Initialized, Role: FOLLOWER
I20240502 21:29:41.136354 16131 consensus_queue.cc:260] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:41.138526 16131 ts_tablet_manager.cc:1398] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.000s
I20240502 21:29:41.141034  9263 tablet_copy_service.cc:342] P 525a093ea5bf40dc9bbf9dc39895d670: Request end of tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-4d1a31513bb64578a85adf1f622e5bc0 received from {username='slave'} at 127.27.196.70:52575
I20240502 21:29:41.141237  9263 tablet_copy_service.cc:433] P 525a093ea5bf40dc9bbf9dc39895d670: ending tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-4d1a31513bb64578a85adf1f622e5bc0 on tablet 4d1a31513bb64578a85adf1f622e5bc0 with peer 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:41.171523  9310 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.71:35857, user_credentials={real_user=slave}} blocked reactor thread for 51553.6us
I20240502 21:29:41.206660  9107 consensus_queue.cc:237] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:41.212185 11229 raft_consensus.cc:1261] T 32c306d724954a16be21a34aa4cc0b5f P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:41.213198 15855 consensus_queue.cc:1035] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:41.217362 11365 raft_consensus.cc:1261] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:41.218312 15854 consensus_queue.cc:1035] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:41.222489  9239 raft_consensus.cc:1261] T 32c306d724954a16be21a34aa4cc0b5f P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:41.223179 15855 consensus_queue.cc:1035] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:41.277324  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:41.287668 15854 raft_consensus.cc:2932] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:41.294041  9242 raft_consensus.cc:2932] T 32c306d724954a16be21a34aa4cc0b5f P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:41.303496 11229 raft_consensus.cc:2932] T 32c306d724954a16be21a34aa4cc0b5f P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:41.304626  8958 catalog_manager.cc:5484] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index 5 to 6, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:41.335712 11364 raft_consensus.cc:2932] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:41.350314  9490 tablet_service.cc:1514] Processing DeleteTablet for tablet 32c306d724954a16be21a34aa4cc0b5f with delete_type TABLET_DATA_TOMBSTONED (TS 19f162e164854f2da623a4bb8f9e6bf6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54670
I20240502 21:29:41.368119 16231 tablet_replica.cc:330] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6: stopping tablet replica
I20240502 21:29:41.368584 16231 raft_consensus.cc:2229] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:41.369020 16231 raft_consensus.cc:2258] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:41.370210 16231 ts_tablet_manager.cc:1875] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 21:29:41.379738  9039 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 70761us
I20240502 21:29:41.391577  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:29:41.405241 11229 raft_consensus.cc:1203] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 21:29:41.426872  9221 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:41.444844 11209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:41.447616 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:41.480533 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:41.490492 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:41.524016 11229 consensus_queue.cc:237] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:41.528301  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:41.552147 16231 ts_tablet_manager.cc:1888] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:29:41.552467 16231 log.cc:1196] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/wals/32c306d724954a16be21a34aa4cc0b5f
I20240502 21:29:41.563868  8945 catalog_manager.cc:4830] TS 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): tablet 32c306d724954a16be21a34aa4cc0b5f (table rebalance_test_table_5 [id=e8eef05f850243d0bdc7e1899f8c6139]) successfully deleted
I20240502 21:29:41.581637 11364 raft_consensus.cc:1261] T 07d693b680a54e12871121737e3da744 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Refusing update from remote peer 6ef0beaf2f6941afb6ea9f9f274517fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 21:29:41.582562 16054 consensus_queue.cc:1035] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:41.592283 11094 raft_consensus.cc:1261] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Refusing update from remote peer 6ef0beaf2f6941afb6ea9f9f274517fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 21:29:41.595605  9378 raft_consensus.cc:1261] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Refusing update from remote peer 6ef0beaf2f6941afb6ea9f9f274517fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 21:29:41.596509 16054 consensus_queue.cc:1035] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:41.597280 16054 consensus_queue.cc:1035] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:41.629266  9507 raft_consensus.cc:1261] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Refusing update from remote peer 6ef0beaf2f6941afb6ea9f9f274517fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 21:29:41.636188 16262 consensus_queue.cc:1035] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:41.645625 16261 raft_consensus.cc:2932] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:41.647167  9378 raft_consensus.cc:2932] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:41.655911  8958 catalog_manager.cc:5484] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: config changed from index 6 to 8, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New cstate: current_term: 2 leader_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:41.663388  9221 tablet_service.cc:1514] Processing DeleteTablet for tablet 07d693b680a54e12871121737e3da744 with delete_type TABLET_DATA_TOMBSTONED (TS 525a093ea5bf40dc9bbf9dc39895d670 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:44408
I20240502 21:29:41.669421  9507 raft_consensus.cc:2932] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:41.675400 16267 tablet_replica.cc:330] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670: stopping tablet replica
I20240502 21:29:41.675882 16267 raft_consensus.cc:2229] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:41.676321 16267 raft_consensus.cc:2258] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:41.677521 16267 ts_tablet_manager.cc:1875] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:41.689455 16267 ts_tablet_manager.cc:1888] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 21:29:41.689687 16267 log.cc:1196] T 07d693b680a54e12871121737e3da744 P 525a093ea5bf40dc9bbf9dc39895d670: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/wals/07d693b680a54e12871121737e3da744
I20240502 21:29:41.694160  8949 catalog_manager.cc:4830] TS 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): tablet 07d693b680a54e12871121737e3da744 (table rebalance_test_table_1 [id=f52c6c3804274777abfa00bb33ae9b16]) successfully deleted
I20240502 21:29:41.699206 11364 raft_consensus.cc:2932] T 07d693b680a54e12871121737e3da744 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:41.704284 11094 raft_consensus.cc:2932] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:41.795912  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:41.817342 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:41.886647 11209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:41.892392  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:29:41.912935  9221 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:41.939867 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:41.941056 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:41.993734  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:42.351413  9507 consensus_queue.cc:237] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:42.357913 11364 raft_consensus.cc:1261] T a2f3edaf1c874cefb820f027f90d1185 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:42.358670 15937 consensus_queue.cc:1035] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:42.361984 11495 raft_consensus.cc:1261] T a2f3edaf1c874cefb820f027f90d1185 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:42.362895 15937 consensus_queue.cc:1035] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:42.363209  9378 raft_consensus.cc:1261] T a2f3edaf1c874cefb820f027f90d1185 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:42.364039 15937 consensus_queue.cc:1035] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:42.392045 11209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:42.394210  9221 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:42.402927 15937 raft_consensus.cc:2932] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:42.406697  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:42.408371 11365 raft_consensus.cc:2932] T a2f3edaf1c874cefb820f027f90d1185 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:42.419910 11496 raft_consensus.cc:2932] T a2f3edaf1c874cefb820f027f90d1185 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:42.440706 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:42.442116  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:29:42.465848 11476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 21:29:42.493698 16373 log.cc:924] Time spent T a2f3edaf1c874cefb820f027f90d1185 P f2fdd596a8b44bbc848fe40f9f07fbe0: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20240502 21:29:42.498365  8962 catalog_manager.cc:5484] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: config changed from index 5 to 6, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New cstate: current_term: 1 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:42.502818  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:42.506237 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:42.514362  9239 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "94d2cdf8ae6941668b5f680c558656e8"
dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:53836
I20240502 21:29:42.514670  9239 raft_consensus.cc:595] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:29:42.544512  9378 raft_consensus.cc:2932] T a2f3edaf1c874cefb820f027f90d1185 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:42.602676  9085 tablet_service.cc:1514] Processing DeleteTablet for tablet a2f3edaf1c874cefb820f027f90d1185 with delete_type TABLET_DATA_TOMBSTONED (TS 5543ddab5b484642b139b7c848932a58 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47814
I20240502 21:29:42.618909 16397 tablet_replica.cc:330] T a2f3edaf1c874cefb820f027f90d1185 P 5543ddab5b484642b139b7c848932a58: stopping tablet replica
I20240502 21:29:42.619354 16397 raft_consensus.cc:2229] T a2f3edaf1c874cefb820f027f90d1185 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:42.619647 16397 raft_consensus.cc:2258] T a2f3edaf1c874cefb820f027f90d1185 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:42.620504 16397 ts_tablet_manager.cc:1875] T a2f3edaf1c874cefb820f027f90d1185 P 5543ddab5b484642b139b7c848932a58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:42.634157 16397 ts_tablet_manager.cc:1888] T a2f3edaf1c874cefb820f027f90d1185 P 5543ddab5b484642b139b7c848932a58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:29:42.635470 16397 log.cc:1196] T a2f3edaf1c874cefb820f027f90d1185 P 5543ddab5b484642b139b7c848932a58: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/wals/a2f3edaf1c874cefb820f027f90d1185
I20240502 21:29:42.636600  8945 catalog_manager.cc:4830] TS 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): tablet a2f3edaf1c874cefb820f027f90d1185 (table rebalance_test_table_8 [id=5802da0404e74a07bca290cdc59b279b]) successfully deleted
I20240502 21:29:42.712823 16003 raft_consensus.cc:979] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670: : Instructing follower e0de0545fa0547d3929111e990e6d1b2 to start an election
I20240502 21:29:42.713030 16003 raft_consensus.cc:1067] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Signalling peer e0de0545fa0547d3929111e990e6d1b2 to start an election
I20240502 21:29:42.714347 11496 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "94d2cdf8ae6941668b5f680c558656e8"
dest_uuid: "e0de0545fa0547d3929111e990e6d1b2"
 from {username='slave'} at 127.27.196.66:44945
I20240502 21:29:42.714700 11496 raft_consensus.cc:482] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:42.714879 11496 raft_consensus.cc:3037] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:42.717867 11496 raft_consensus.cc:504] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:42.720762  9105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d2cdf8ae6941668b5f680c558656e8" candidate_uuid: "e0de0545fa0547d3929111e990e6d1b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "5543ddab5b484642b139b7c848932a58"
I20240502 21:29:42.721009  9105 raft_consensus.cc:3037] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:42.724535  9105 raft_consensus.cc:2454] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0de0545fa0547d3929111e990e6d1b2 in term 2.
I20240502 21:29:42.725348 11429 leader_election.cc:302] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5543ddab5b484642b139b7c848932a58, e0de0545fa0547d3929111e990e6d1b2; no voters: 
I20240502 21:29:42.725838 16378 raft_consensus.cc:2781] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:42.731526  9239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d2cdf8ae6941668b5f680c558656e8" candidate_uuid: "e0de0545fa0547d3929111e990e6d1b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670"
I20240502 21:29:42.731796  9239 raft_consensus.cc:3032] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:42.731871  9239 raft_consensus.cc:729] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Running, Role: LEADER
I20240502 21:29:42.732205  9239 consensus_queue.cc:260] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:42.732585  9239 raft_consensus.cc:3037] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:42.735980  9239 raft_consensus.cc:2454] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0de0545fa0547d3929111e990e6d1b2 in term 2.
I20240502 21:29:42.764827 11496 leader_election.cc:288] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 election: Requested vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:29:42.771545 16378 raft_consensus.cc:686] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 [term 2 LEADER]: Becoming Leader. State: Replica: e0de0545fa0547d3929111e990e6d1b2, State: Running, Role: LEADER
I20240502 21:29:42.772368 16378 consensus_queue.cc:237] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:42.785723  8963 catalog_manager.cc:5484] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 reported cstate change: term changed from 1 to 2, leader changed from 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) to e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72). New cstate: current_term: 2 leader_uuid: "e0de0545fa0547d3929111e990e6d1b2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:42.922443  9105 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cffcb4f7f7754262beeb06168097f6dc"
dest_uuid: "5543ddab5b484642b139b7c848932a58"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:41920
I20240502 21:29:42.922706  9105 raft_consensus.cc:595] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:29:42.962661 16406 raft_consensus.cc:979] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58: : Instructing follower f2fdd596a8b44bbc848fe40f9f07fbe0 to start an election
I20240502 21:29:42.962857 16406 raft_consensus.cc:1067] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Signalling peer f2fdd596a8b44bbc848fe40f9f07fbe0 to start an election
I20240502 21:29:42.964020  9378 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cffcb4f7f7754262beeb06168097f6dc"
dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
 from {username='slave'} at 127.27.196.65:38501
I20240502 21:29:42.964282  9378 raft_consensus.cc:482] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:42.964394  9378 raft_consensus.cc:3037] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:42.977855  9378 raft_consensus.cc:504] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:42.981227  9239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffcb4f7f7754262beeb06168097f6dc" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670"
I20240502 21:29:42.981494  9239 raft_consensus.cc:3037] T cffcb4f7f7754262beeb06168097f6dc P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:42.990115  9239 raft_consensus.cc:2454] T cffcb4f7f7754262beeb06168097f6dc P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 2.
I20240502 21:29:42.999850 11496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffcb4f7f7754262beeb06168097f6dc" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e0de0545fa0547d3929111e990e6d1b2"
I20240502 21:29:43.000262 11496 raft_consensus.cc:3037] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:43.005029  9378 leader_election.cc:288] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 2 election: Requested vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467)
I20240502 21:29:43.006716 11496 raft_consensus.cc:2454] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 2.
I20240502 21:29:43.007632  9313 leader_election.cc:302] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 525a093ea5bf40dc9bbf9dc39895d670, e0de0545fa0547d3929111e990e6d1b2, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:29:43.009588 16453 raft_consensus.cc:2781] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:43.009734 16453 raft_consensus.cc:686] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 LEADER]: Becoming Leader. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Running, Role: LEADER
I20240502 21:29:43.010804 16453 consensus_queue.cc:237] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:43.012909  9105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffcb4f7f7754262beeb06168097f6dc" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5543ddab5b484642b139b7c848932a58"
I20240502 21:29:43.013149  9105 raft_consensus.cc:3032] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:43.013237  9105 raft_consensus.cc:729] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Running, Role: LEADER
I20240502 21:29:43.013499  9105 consensus_queue.cc:260] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:43.013978  9105 raft_consensus.cc:3037] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:43.022465  9105 raft_consensus.cc:2454] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 2.
I20240502 21:29:43.050376  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:43.071167  8962 catalog_manager.cc:5484] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: term changed from 1 to 2, leader changed from 5543ddab5b484642b139b7c848932a58 (127.27.196.65) to f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67). New cstate: current_term: 2 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:43.145411  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:43.146914 11209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:43.165388 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:43.171492  9105 raft_consensus.cc:1261] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Refusing update from remote peer e0de0545fa0547d3929111e990e6d1b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:29:43.172550 16378 consensus_queue.cc:1035] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:43.204509 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:43.265434  9221 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:29:43.277581 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:43.303366  9239 raft_consensus.cc:1261] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Refusing update from remote peer e0de0545fa0547d3929111e990e6d1b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:29:43.304519 16415 consensus_queue.cc:1035] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:43.308936 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:43.360270  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:43.379932 11476 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:43.383531 11209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:43.418532 11476 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:43.439567  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:43.497437  9356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:43.519958 11496 raft_consensus.cc:1261] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:29:43.520932 16453 consensus_queue.cc:1035] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:43.600278  9356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:43.615003  9221 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:29:43.625284  9107 raft_consensus.cc:1261] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:29:43.626132 16457 consensus_queue.cc:1035] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:43.630115  9242 raft_consensus.cc:1261] T cffcb4f7f7754262beeb06168097f6dc P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:29:43.631879 16453 consensus_queue.cc:1035] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:43.835844  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:43.899065 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:43.945245  9221 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:29:43.978439  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:44.005095 11209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:44.054140 11476 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:44.068011 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:44.119573  9356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:44.209656  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:44.215499  9221 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:29:44.253053 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:44.261869  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:44.332753  9356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:44.351907 11476 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:44.363250 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:44.363626 11209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:44.386337  9239 consensus_queue.cc:237] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:29:44.449748  9507 raft_consensus.cc:1261] T 58c7004ceb044f5fa9dec54910046b69 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:44.450596 15027 consensus_queue.cc:1035] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:44.524612 16616 raft_consensus.cc:2932] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } }
I20240502 21:29:44.525218  9507 raft_consensus.cc:2932] T 58c7004ceb044f5fa9dec54910046b69 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } }
I20240502 21:29:44.534045  8961 catalog_manager.cc:5484] T 58c7004ceb044f5fa9dec54910046b69 P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: config changed from index 5 to 6, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } }
I20240502 21:29:44.682593  9356 tablet_service.cc:1514] Processing DeleteTablet for tablet 58c7004ceb044f5fa9dec54910046b69 with delete_type TABLET_DATA_TOMBSTONED (TS f2fdd596a8b44bbc848fe40f9f07fbe0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55106
I20240502 21:29:44.683382  9239 consensus_queue.cc:237] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:44.687362  9507 raft_consensus.cc:1261] T 58c7004ceb044f5fa9dec54910046b69 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:44.690703 16636 tablet_replica.cc:330] T 58c7004ceb044f5fa9dec54910046b69 P f2fdd596a8b44bbc848fe40f9f07fbe0: stopping tablet replica
I20240502 21:29:44.691026 16636 raft_consensus.cc:2229] T 58c7004ceb044f5fa9dec54910046b69 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:44.692847 16636 raft_consensus.cc:2258] T 58c7004ceb044f5fa9dec54910046b69 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:44.693820 16636 ts_tablet_manager.cc:1875] T 58c7004ceb044f5fa9dec54910046b69 P f2fdd596a8b44bbc848fe40f9f07fbe0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:44.694481 16616 consensus_queue.cc:1035] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 21:29:44.695643  9175 consensus_peers.cc:485] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467): Couldn't send request to peer e0de0545fa0547d3929111e990e6d1b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 58c7004ceb044f5fa9dec54910046b69. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:44.708503 16616 raft_consensus.cc:2932] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:44.714568  9507 raft_consensus.cc:2932] T 58c7004ceb044f5fa9dec54910046b69 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:44.727306 16636 ts_tablet_manager.cc:1888] T 58c7004ceb044f5fa9dec54910046b69 P f2fdd596a8b44bbc848fe40f9f07fbe0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:29:44.727502 16636 log.cc:1196] T 58c7004ceb044f5fa9dec54910046b69 P f2fdd596a8b44bbc848fe40f9f07fbe0: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/wals/58c7004ceb044f5fa9dec54910046b69
I20240502 21:29:44.729787  8948 catalog_manager.cc:4830] TS f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): tablet 58c7004ceb044f5fa9dec54910046b69 (table rebalance_test_table_1 [id=f52c6c3804274777abfa00bb33ae9b16]) successfully deleted
I20240502 21:29:44.738649  8963 catalog_manager.cc:5484] T 58c7004ceb044f5fa9dec54910046b69 P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: config changed from index 6 to 7, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:44.741300  8949 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 58c7004ceb044f5fa9dec54910046b69 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 21:29:44.744710  9378 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a7c6b881c9ef4cdca787f065778e526f"
dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:43852
I20240502 21:29:44.745018  9378 raft_consensus.cc:595] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:29:44.764860  9378 consensus_queue.cc:237] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:44.780727 11365 raft_consensus.cc:1261] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:44.781980 16453 consensus_queue.cc:1035] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:44.782351 11496 raft_consensus.cc:1261] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:44.799822  9239 raft_consensus.cc:1261] T 41ee481a3ece47a0905e93c66dbf949d P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:44.800959 16453 consensus_queue.cc:1035] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:44.805685  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:44.809425 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:44.829998 16453 consensus_queue.cc:1035] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:44.838640  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:44.941013 16677 raft_consensus.cc:2932] T 41ee481a3ece47a0905e93c66dbf949d P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:44.942538  9239 raft_consensus.cc:2932] T 41ee481a3ece47a0905e93c66dbf949d P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:44.948026 16678 raft_consensus.cc:979] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0: : Instructing follower e0de0545fa0547d3929111e990e6d1b2 to start an election
I20240502 21:29:44.948305 16457 raft_consensus.cc:1067] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Signalling peer e0de0545fa0547d3929111e990e6d1b2 to start an election
I20240502 21:29:44.949752 11495 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a7c6b881c9ef4cdca787f065778e526f"
dest_uuid: "e0de0545fa0547d3929111e990e6d1b2"
 from {username='slave'} at 127.27.196.67:39199
I20240502 21:29:44.950083 11495 raft_consensus.cc:482] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:44.950233 11495 raft_consensus.cc:3037] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:44.956521 11495 raft_consensus.cc:504] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:44.956655  8963 catalog_manager.cc:5484] T 41ee481a3ece47a0905e93c66dbf949d P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: config changed from index 5 to 6, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:44.958613 11495 leader_election.cc:288] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 election: Requested vote from peers f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:29:44.959237  9378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c6b881c9ef4cdca787f065778e526f" candidate_uuid: "e0de0545fa0547d3929111e990e6d1b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
I20240502 21:29:44.959600  9378 raft_consensus.cc:3032] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:44.959692  9378 raft_consensus.cc:729] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Running, Role: LEADER
I20240502 21:29:44.960073  9378 consensus_queue.cc:260] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:44.960503  9378 raft_consensus.cc:3037] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:44.978304  9378 raft_consensus.cc:2454] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0de0545fa0547d3929111e990e6d1b2 in term 2.
I20240502 21:29:44.979466 11431 leader_election.cc:302] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:29:44.980072 16378 raft_consensus.cc:2781] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:44.985944  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:44.997908 16378 raft_consensus.cc:686] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [term 2 LEADER]: Becoming Leader. State: Replica: e0de0545fa0547d3929111e990e6d1b2, State: Running, Role: LEADER
I20240502 21:29:44.998610 16378 consensus_queue.cc:237] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:45.023643  9242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c6b881c9ef4cdca787f065778e526f" candidate_uuid: "e0de0545fa0547d3929111e990e6d1b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670"
I20240502 21:29:45.024034  9242 raft_consensus.cc:3037] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:45.028837  9242 raft_consensus.cc:2454] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0de0545fa0547d3929111e990e6d1b2 in term 2.
I20240502 21:29:45.067726 11365 raft_consensus.cc:2932] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:45.068890  8961 catalog_manager.cc:5484] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 reported cstate change: term changed from 1 to 2, leader changed from f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) to e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72). New cstate: current_term: 2 leader_uuid: "e0de0545fa0547d3929111e990e6d1b2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:45.084872 11209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:45.108072  9085 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ee481a3ece47a0905e93c66dbf949d with delete_type TABLET_DATA_TOMBSTONED (TS 5543ddab5b484642b139b7c848932a58 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47814
I20240502 21:29:45.110028 16685 tablet_replica.cc:330] T 41ee481a3ece47a0905e93c66dbf949d P 5543ddab5b484642b139b7c848932a58: stopping tablet replica
I20240502 21:29:45.110375 16685 raft_consensus.cc:2229] T 41ee481a3ece47a0905e93c66dbf949d P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:45.110848  9221 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:29:45.111007 16685 raft_consensus.cc:2258] T 41ee481a3ece47a0905e93c66dbf949d P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:45.115343 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:45.112035 16685 ts_tablet_manager.cc:1875] T 41ee481a3ece47a0905e93c66dbf949d P 5543ddab5b484642b139b7c848932a58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:45.119096 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:45.122941 11496 raft_consensus.cc:2932] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:45.132355 16685 ts_tablet_manager.cc:1888] T 41ee481a3ece47a0905e93c66dbf949d P 5543ddab5b484642b139b7c848932a58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:29:45.133833 16685 log.cc:1196] T 41ee481a3ece47a0905e93c66dbf949d P 5543ddab5b484642b139b7c848932a58: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/wals/41ee481a3ece47a0905e93c66dbf949d
I20240502 21:29:45.135417  8945 catalog_manager.cc:4830] TS 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): tablet 41ee481a3ece47a0905e93c66dbf949d (table rebalance_test_table_0 [id=445d2e6ec7154b80abec5fff346b8f5e]) successfully deleted
I20240502 21:29:45.299746 16702 ts_tablet_manager.cc:902] T 58c7004ceb044f5fa9dec54910046b69 P e0de0545fa0547d3929111e990e6d1b2: Initiating tablet copy from peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:29:45.388012 16702 tablet_copy_client.cc:308] T 58c7004ceb044f5fa9dec54910046b69 P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:39071
I20240502 21:29:45.389928  9263 tablet_copy_service.cc:140] P 525a093ea5bf40dc9bbf9dc39895d670: Received BeginTabletCopySession request for tablet 58c7004ceb044f5fa9dec54910046b69 from peer e0de0545fa0547d3929111e990e6d1b2 ({username='slave'} at 127.27.196.72:35369)
I20240502 21:29:45.390223  9263 tablet_copy_service.cc:161] P 525a093ea5bf40dc9bbf9dc39895d670: Beginning new tablet copy session on tablet 58c7004ceb044f5fa9dec54910046b69 from peer e0de0545fa0547d3929111e990e6d1b2 at {username='slave'} at 127.27.196.72:35369: session id = e0de0545fa0547d3929111e990e6d1b2-58c7004ceb044f5fa9dec54910046b69
I20240502 21:29:45.397704  9263 tablet_copy_source_session.cc:205] T 58c7004ceb044f5fa9dec54910046b69 P 525a093ea5bf40dc9bbf9dc39895d670: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:45.400650 16702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c7004ceb044f5fa9dec54910046b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:45.434919 16702 tablet_copy_client.cc:783] T 58c7004ceb044f5fa9dec54910046b69 P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:45.437585 16702 tablet_copy_client.cc:647] T 58c7004ceb044f5fa9dec54910046b69 P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:45.474638 16702 tablet_copy_client.cc:523] T 58c7004ceb044f5fa9dec54910046b69 P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:45.518188 16702 tablet_bootstrap.cc:492] T 58c7004ceb044f5fa9dec54910046b69 P e0de0545fa0547d3929111e990e6d1b2: Bootstrap starting.
I20240502 21:29:45.535015  9242 raft_consensus.cc:1261] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Refusing update from remote peer e0de0545fa0547d3929111e990e6d1b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:29:45.535979 16378 consensus_queue.cc:1035] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:45.571801  9378 raft_consensus.cc:1261] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Refusing update from remote peer e0de0545fa0547d3929111e990e6d1b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:29:45.573025 16378 consensus_queue.cc:1035] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:45.770576 16702 tablet_bootstrap.cc:492] T 58c7004ceb044f5fa9dec54910046b69 P e0de0545fa0547d3929111e990e6d1b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:45.775779 16702 tablet_bootstrap.cc:492] T 58c7004ceb044f5fa9dec54910046b69 P e0de0545fa0547d3929111e990e6d1b2: Bootstrap complete.
I20240502 21:29:45.776506 16702 ts_tablet_manager.cc:1367] T 58c7004ceb044f5fa9dec54910046b69 P e0de0545fa0547d3929111e990e6d1b2: Time spent bootstrapping tablet: real 0.258s	user 0.063s	sys 0.019s
I20240502 21:29:45.777923 16702 raft_consensus.cc:348] T 58c7004ceb044f5fa9dec54910046b69 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:45.778422 16702 raft_consensus.cc:729] T 58c7004ceb044f5fa9dec54910046b69 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0de0545fa0547d3929111e990e6d1b2, State: Initialized, Role: FOLLOWER
I20240502 21:29:45.778865 16702 consensus_queue.cc:260] T 58c7004ceb044f5fa9dec54910046b69 P e0de0545fa0547d3929111e990e6d1b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:45.781342 16702 ts_tablet_manager.cc:1398] T 58c7004ceb044f5fa9dec54910046b69 P e0de0545fa0547d3929111e990e6d1b2: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.002s
I20240502 21:29:45.783952  9263 tablet_copy_service.cc:342] P 525a093ea5bf40dc9bbf9dc39895d670: Request end of tablet copy session e0de0545fa0547d3929111e990e6d1b2-58c7004ceb044f5fa9dec54910046b69 received from {username='slave'} at 127.27.196.72:35369
I20240502 21:29:45.784127  9263 tablet_copy_service.cc:433] P 525a093ea5bf40dc9bbf9dc39895d670: ending tablet copy session e0de0545fa0547d3929111e990e6d1b2-58c7004ceb044f5fa9dec54910046b69 on tablet 58c7004ceb044f5fa9dec54910046b69 with peer e0de0545fa0547d3929111e990e6d1b2
I20240502 21:29:45.864313 11496 raft_consensus.cc:1203] T 58c7004ceb044f5fa9dec54910046b69 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 21:29:45.889672 11496 consensus_queue.cc:237] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:45.903069  9107 raft_consensus.cc:1261] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Refusing update from remote peer e0de0545fa0547d3929111e990e6d1b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:29:45.903954 16739 consensus_queue.cc:1035] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:45.933187 16739 raft_consensus.cc:2932] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:45.937551  9105 raft_consensus.cc:2932] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:45.948730  8963 catalog_manager.cc:5484] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 reported cstate change: config changed from index 5 to 7, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New cstate: current_term: 2 leader_uuid: "e0de0545fa0547d3929111e990e6d1b2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:46.108793  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:46.146005  9377 consensus_queue.cc:237] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:46.200943  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:46.206130  9239 raft_consensus.cc:1261] T cffcb4f7f7754262beeb06168097f6dc P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:29:46.206799 11496 raft_consensus.cc:1261] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:29:46.207593 16678 consensus_queue.cc:1035] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:46.210500 16678 consensus_queue.cc:1035] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:46.228189 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:46.256846 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:46.268980  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:46.284494  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:46.345394 16677 raft_consensus.cc:2932] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:46.346304 11495 raft_consensus.cc:2932] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:46.363829  9221 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:29:46.365715  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:46.372337  9242 raft_consensus.cc:2932] T cffcb4f7f7754262beeb06168097f6dc P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:46.404024 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:46.412825 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:46.432349  9221 tablet_service.cc:1514] Processing DeleteTablet for tablet 94d2cdf8ae6941668b5f680c558656e8 with delete_type TABLET_DATA_TOMBSTONED (TS 525a093ea5bf40dc9bbf9dc39895d670 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44408
I20240502 21:29:46.432714  8961 catalog_manager.cc:5484] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: config changed from index 4 to 6, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New cstate: current_term: 2 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:46.471305  9085 tablet_service.cc:1514] Processing DeleteTablet for tablet cffcb4f7f7754262beeb06168097f6dc with delete_type TABLET_DATA_TOMBSTONED (TS 5543ddab5b484642b139b7c848932a58 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47814
I20240502 21:29:46.506917 11495 consensus_queue.cc:237] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:46.508158 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:46.514595  9105 raft_consensus.cc:1261] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Refusing update from remote peer e0de0545fa0547d3929111e990e6d1b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 21:29:46.516117 11209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:46.515991 16378 consensus_queue.cc:1035] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20240502 21:29:46.539706 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:46.556356  9219 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:29:46.558108 16884 tablet_replica.cc:330] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670: stopping tablet replica
I20240502 21:29:46.558490 16884 raft_consensus.cc:2229] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:46.558807 16884 raft_consensus.cc:2258] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:46.559895 16884 ts_tablet_manager.cc:1875] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:46.567850 16887 tablet_replica.cc:330] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58: stopping tablet replica
I20240502 21:29:46.568320 16887 raft_consensus.cc:2229] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:46.568608 16887 raft_consensus.cc:2258] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:46.569700 16887 ts_tablet_manager.cc:1875] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:46.581670  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:46.588344 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:46.597286 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:46.597746 16898 raft_consensus.cc:2932] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:46.619585  9105 raft_consensus.cc:2932] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:46.629029  8961 catalog_manager.cc:5484] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index 7 to 8, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New cstate: current_term: 2 leader_uuid: "e0de0545fa0547d3929111e990e6d1b2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:46.629765  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:46.688045 11209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:46.692519 16887 ts_tablet_manager.cc:1888] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 21:29:46.692685 16887 log.cc:1196] T cffcb4f7f7754262beeb06168097f6dc P 5543ddab5b484642b139b7c848932a58: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/wals/cffcb4f7f7754262beeb06168097f6dc
I20240502 21:29:46.693764  8945 catalog_manager.cc:4830] TS 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): tablet cffcb4f7f7754262beeb06168097f6dc (table rebalance_test_table_4 [id=1b17408dbab9492bba0ed50efd6ef43f]) successfully deleted
I20240502 21:29:46.696847 16884 ts_tablet_manager.cc:1888] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 21:29:46.697000 16884 log.cc:1196] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/wals/94d2cdf8ae6941668b5f680c558656e8
I20240502 21:29:46.698086  8949 catalog_manager.cc:4830] TS 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): tablet 94d2cdf8ae6941668b5f680c558656e8 (table rebalance_test_table_7 [id=ef412fa8fe9249729cb99879a5c8343a]) successfully deleted
I20240502 21:29:46.718408  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:29:46.720221 11429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.65:42177, user_credentials={real_user=slave}} blocked reactor thread for 95444.6us
W20240502 21:29:46.721132 11162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.67:43203, user_credentials={real_user=slave}} blocked reactor thread for 78788.2us
I20240502 21:29:46.722770  9221 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:29:46.723349  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:46.760190  8949 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 94d2cdf8ae6941668b5f680c558656e8 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20240502 21:29:46.761171 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 94d2cdf8ae6941668b5f680c558656e8. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:46.780200 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:46.787922 11209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:47.180939 16944 ts_tablet_manager.cc:902] T 94d2cdf8ae6941668b5f680c558656e8 P 6ef0beaf2f6941afb6ea9f9f274517fe: Initiating tablet copy from peer e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467)
I20240502 21:29:47.230962  9239 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4d1a31513bb64578a85adf1f622e5bc0"
dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:53972
I20240502 21:29:47.231302  9239 raft_consensus.cc:595] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:29:47.309125 16944 tablet_copy_client.cc:308] T 94d2cdf8ae6941668b5f680c558656e8 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.72:40467
I20240502 21:29:47.383253 16849 raft_consensus.cc:979] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670: : Instructing follower 6ef0beaf2f6941afb6ea9f9f274517fe to start an election
I20240502 21:29:47.383430 16849 raft_consensus.cc:1067] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Signalling peer 6ef0beaf2f6941afb6ea9f9f274517fe to start an election
I20240502 21:29:47.385139 11228 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4d1a31513bb64578a85adf1f622e5bc0"
dest_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe"
 from {username='slave'} at 127.27.196.66:33219
I20240502 21:29:47.385458 11228 raft_consensus.cc:482] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:47.385596 11228 raft_consensus.cc:3037] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:47.390442 11228 raft_consensus.cc:504] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:47.393080  9105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1a31513bb64578a85adf1f622e5bc0" candidate_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "5543ddab5b484642b139b7c848932a58"
I20240502 21:29:47.393389  9105 raft_consensus.cc:3037] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:47.396982  9105 raft_consensus.cc:2454] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef0beaf2f6941afb6ea9f9f274517fe in term 2.
I20240502 21:29:47.397761 11160 leader_election.cc:302] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5543ddab5b484642b139b7c848932a58, 6ef0beaf2f6941afb6ea9f9f274517fe; no voters: 
I20240502 21:29:47.398159 16261 raft_consensus.cc:2781] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:47.428304  9239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1a31513bb64578a85adf1f622e5bc0" candidate_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670"
I20240502 21:29:47.428740  9239 raft_consensus.cc:3032] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:47.428838  9239 raft_consensus.cc:729] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Running, Role: LEADER
I20240502 21:29:47.429234  9239 consensus_queue.cc:260] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:47.429737  9239 raft_consensus.cc:3037] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:47.436280  9239 raft_consensus.cc:2454] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef0beaf2f6941afb6ea9f9f274517fe in term 2.
I20240502 21:29:47.432094 11228 leader_election.cc:288] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [CANDIDATE]: Term 2 election: Requested vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:29:47.445731 11517 tablet_copy_service.cc:140] P e0de0545fa0547d3929111e990e6d1b2: Received BeginTabletCopySession request for tablet 94d2cdf8ae6941668b5f680c558656e8 from peer 6ef0beaf2f6941afb6ea9f9f274517fe ({username='slave'} at 127.27.196.70:49249)
I20240502 21:29:47.446097 11517 tablet_copy_service.cc:161] P e0de0545fa0547d3929111e990e6d1b2: Beginning new tablet copy session on tablet 94d2cdf8ae6941668b5f680c558656e8 from peer 6ef0beaf2f6941afb6ea9f9f274517fe at {username='slave'} at 127.27.196.70:49249: session id = 6ef0beaf2f6941afb6ea9f9f274517fe-94d2cdf8ae6941668b5f680c558656e8
I20240502 21:29:47.448290  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:47.452001 11517 tablet_copy_source_session.cc:205] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:47.456440 16944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94d2cdf8ae6941668b5f680c558656e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:47.466243 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:47.473083  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:47.474117 16944 tablet_copy_client.cc:783] T 94d2cdf8ae6941668b5f680c558656e8 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:47.474565 16944 tablet_copy_client.cc:647] T 94d2cdf8ae6941668b5f680c558656e8 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:47.488695 16944 tablet_copy_client.cc:523] T 94d2cdf8ae6941668b5f680c558656e8 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:47.504639  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:47.507952 16944 tablet_bootstrap.cc:492] T 94d2cdf8ae6941668b5f680c558656e8 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap starting.
I20240502 21:29:47.519080 16261 raft_consensus.cc:686] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 LEADER]: Becoming Leader. State: Replica: 6ef0beaf2f6941afb6ea9f9f274517fe, State: Running, Role: LEADER
I20240502 21:29:47.519491 16261 consensus_queue.cc:237] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
W20240502 21:29:47.521312  9313 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.66:39071, user_credentials={real_user=slave}} blocked reactor thread for 146127us
I20240502 21:29:47.531877 11209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:47.533541  8962 catalog_manager.cc:5484] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe reported cstate change: term changed from 1 to 2, leader changed from 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) to 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70). New cstate: current_term: 2 leader_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:47.535346 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:47.563398 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:47.592258  9239 consensus_queue.cc:237] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:47.626466 11364 raft_consensus.cc:1261] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:47.627318 16624 consensus_queue.cc:1035] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:47.639752  9377 raft_consensus.cc:1261] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:47.650408 16849 consensus_queue.cc:1035] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:47.663372 16849 raft_consensus.cc:2932] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:47.668449 11365 raft_consensus.cc:2932] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:47.672307  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:47.686611  8961 catalog_manager.cc:5484] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: config changed from index 4 to 5, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:47.691792 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:47.695271 11209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:47.733705  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:47.739152  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:47.741187  9378 raft_consensus.cc:2932] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:47.742043  9490 tablet_service.cc:1514] Processing DeleteTablet for tablet 87f35aff2b104c01a9d9d48a7e9c9b23 with delete_type TABLET_DATA_TOMBSTONED (TS 19f162e164854f2da623a4bb8f9e6bf6 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54670
I20240502 21:29:47.758939 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:47.766885 17049 tablet_replica.cc:330] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 19f162e164854f2da623a4bb8f9e6bf6: stopping tablet replica
I20240502 21:29:47.768153 17049 raft_consensus.cc:2229] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:47.769379 17049 raft_consensus.cc:2258] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:47.784888 17049 ts_tablet_manager.cc:1875] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 19f162e164854f2da623a4bb8f9e6bf6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:47.800225 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:47.809850  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:47.815645  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:47.884747 17049 ts_tablet_manager.cc:1888] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 19f162e164854f2da623a4bb8f9e6bf6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:29:47.908298 17049 log.cc:1196] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 19f162e164854f2da623a4bb8f9e6bf6: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/wals/87f35aff2b104c01a9d9d48a7e9c9b23
I20240502 21:29:47.912261  8945 catalog_manager.cc:4830] TS 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): tablet 87f35aff2b104c01a9d9d48a7e9c9b23 (table rebalance_test_table_7 [id=ef412fa8fe9249729cb99879a5c8343a]) successfully deleted
I20240502 21:29:47.914775  9239 raft_consensus.cc:1261] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Refusing update from remote peer 6ef0beaf2f6941afb6ea9f9f274517fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 21:29:47.915791 16261 consensus_queue.cc:1035] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:47.933388 11495 consensus_queue.cc:237] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:47.964079  9242 raft_consensus.cc:1261] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Refusing update from remote peer e0de0545fa0547d3929111e990e6d1b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:29:47.965420 16898 consensus_queue.cc:1035] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:47.993453  9105 raft_consensus.cc:1261] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Refusing update from remote peer 6ef0beaf2f6941afb6ea9f9f274517fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 21:29:47.996932 16261 consensus_queue.cc:1035] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:48.053951  9242 raft_consensus.cc:2932] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:48.061601  8962 catalog_manager.cc:5484] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: config changed from index 5 to 7, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New cstate: current_term: 2 leader_uuid: "e0de0545fa0547d3929111e990e6d1b2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:48.067497  9356 tablet_service.cc:1514] Processing DeleteTablet for tablet a7c6b881c9ef4cdca787f065778e526f with delete_type TABLET_DATA_TOMBSTONED (TS f2fdd596a8b44bbc848fe40f9f07fbe0 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55106
I20240502 21:29:48.052321 17067 raft_consensus.cc:2932] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:48.083011 17073 tablet_replica.cc:330] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0: stopping tablet replica
I20240502 21:29:48.083303 17073 raft_consensus.cc:2229] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:48.090564 17073 raft_consensus.cc:2258] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:48.093910 11495 consensus_queue.cc:237] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:48.099577  9242 raft_consensus.cc:1261] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Refusing update from remote peer e0de0545fa0547d3929111e990e6d1b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 21:29:48.100345 16898 consensus_queue.cc:1035] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:48.104996  9239 raft_consensus.cc:2932] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:48.109850 17073 ts_tablet_manager.cc:1875] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:48.119243  8961 catalog_manager.cc:5484] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: config changed from index 7 to 8, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New cstate: current_term: 2 leader_uuid: "e0de0545fa0547d3929111e990e6d1b2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:48.123572 17067 raft_consensus.cc:2932] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:48.141023  8949 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet a7c6b881c9ef4cdca787f065778e526f with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20240502 21:29:48.153198 11430 consensus_peers.cc:485] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Couldn't send request to peer d10e9e156ba74a97a221c85b68ab8bc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7c6b881c9ef4cdca787f065778e526f. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:48.166607 17073 ts_tablet_manager.cc:1888] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 21:29:48.166807 17073 log.cc:1196] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/wals/a7c6b881c9ef4cdca787f065778e526f
I20240502 21:29:48.169123  8948 catalog_manager.cc:4830] TS f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): tablet a7c6b881c9ef4cdca787f065778e526f (table rebalance_test_table_6 [id=265d5656d7c048aa9795a906661b8ea8]) successfully deleted
I20240502 21:29:48.173134 16944 tablet_bootstrap.cc:492] T 94d2cdf8ae6941668b5f680c558656e8 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:48.173915 16944 tablet_bootstrap.cc:492] T 94d2cdf8ae6941668b5f680c558656e8 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap complete.
I20240502 21:29:48.174302 16944 ts_tablet_manager.cc:1367] T 94d2cdf8ae6941668b5f680c558656e8 P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent bootstrapping tablet: real 0.666s	user 0.070s	sys 0.002s
I20240502 21:29:48.179755 16944 raft_consensus.cc:348] T 94d2cdf8ae6941668b5f680c558656e8 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:48.180181 16944 raft_consensus.cc:729] T 94d2cdf8ae6941668b5f680c558656e8 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef0beaf2f6941afb6ea9f9f274517fe, State: Initialized, Role: FOLLOWER
I20240502 21:29:48.180531 16944 consensus_queue.cc:260] T 94d2cdf8ae6941668b5f680c558656e8 P 6ef0beaf2f6941afb6ea9f9f274517fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:48.181316 16944 ts_tablet_manager.cc:1398] T 94d2cdf8ae6941668b5f680c558656e8 P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20240502 21:29:48.185678 11517 tablet_copy_service.cc:342] P e0de0545fa0547d3929111e990e6d1b2: Request end of tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-94d2cdf8ae6941668b5f680c558656e8 received from {username='slave'} at 127.27.196.70:49249
I20240502 21:29:48.186017 11517 tablet_copy_service.cc:433] P e0de0545fa0547d3929111e990e6d1b2: ending tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-94d2cdf8ae6941668b5f680c558656e8 on tablet 94d2cdf8ae6941668b5f680c558656e8 with peer 6ef0beaf2f6941afb6ea9f9f274517fe
I20240502 21:29:48.230888  9377 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "be34ec89876d42b69efe1878c0d9bc19"
dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:43940
I20240502 21:29:48.231303  9377 raft_consensus.cc:595] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:29:48.436535  9239 consensus_queue.cc:237] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:48.460245 11092 raft_consensus.cc:1261] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:48.461166 16849 consensus_queue.cc:1035] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:48.541421  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:48.584443 16849 raft_consensus.cc:2932] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:48.587316 11094 raft_consensus.cc:2932] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:48.591666 11228 raft_consensus.cc:1203] T 94d2cdf8ae6941668b5f680c558656e8 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8]   Dedup: 2.8->[]
I20240502 21:29:48.598196  8962 catalog_manager.cc:5484] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4 reported cstate change: config changed from index 5 to 6, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:48.606431  9490 tablet_service.cc:1514] Processing DeleteTablet for tablet 51936fb1f678458a8795f8246590addb with delete_type TABLET_DATA_TOMBSTONED (TS 19f162e164854f2da623a4bb8f9e6bf6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54670
I20240502 21:29:48.607355 17166 tablet_replica.cc:330] T 51936fb1f678458a8795f8246590addb P 19f162e164854f2da623a4bb8f9e6bf6: stopping tablet replica
I20240502 21:29:48.608080 17166 raft_consensus.cc:2229] T 51936fb1f678458a8795f8246590addb P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:48.609637 17166 raft_consensus.cc:2258] T 51936fb1f678458a8795f8246590addb P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:48.610852 17166 ts_tablet_manager.cc:1875] T 51936fb1f678458a8795f8246590addb P 19f162e164854f2da623a4bb8f9e6bf6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:48.613842  9239 consensus_queue.cc:237] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:48.620407 11094 raft_consensus.cc:1261] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:48.621708 16624 consensus_queue.cc:1035] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 21:29:48.623181  9175 consensus_peers.cc:485] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467): Couldn't send request to peer e0de0545fa0547d3929111e990e6d1b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51936fb1f678458a8795f8246590addb. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:48.628278 17166 ts_tablet_manager.cc:1888] T 51936fb1f678458a8795f8246590addb P 19f162e164854f2da623a4bb8f9e6bf6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:29:48.630564 17166 log.cc:1196] T 51936fb1f678458a8795f8246590addb P 19f162e164854f2da623a4bb8f9e6bf6: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/wals/51936fb1f678458a8795f8246590addb
I20240502 21:29:48.632222  8945 catalog_manager.cc:4830] TS 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): tablet 51936fb1f678458a8795f8246590addb (table rebalance_test_table_2 [id=c0a0ef39a2a246449eb582cc1d993484]) successfully deleted
I20240502 21:29:48.635818 16624 raft_consensus.cc:2932] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:48.641783 11093 raft_consensus.cc:2932] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:48.652315  8962 catalog_manager.cc:5484] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4 reported cstate change: config changed from index 6 to 7, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:48.656958 11209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:48.673529 17164 raft_consensus.cc:979] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0: : Instructing follower 19f162e164854f2da623a4bb8f9e6bf6 to start an election
I20240502 21:29:48.673817 17164 raft_consensus.cc:1067] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Signalling peer 19f162e164854f2da623a4bb8f9e6bf6 to start an election
I20240502 21:29:48.676774  9507 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "be34ec89876d42b69efe1878c0d9bc19"
dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6"
 from {username='slave'} at 127.27.196.67:58623
I20240502 21:29:48.677227  9507 raft_consensus.cc:482] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:48.677404  9507 raft_consensus.cc:3037] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:48.682725  9507 raft_consensus.cc:504] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:48.688479  9105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be34ec89876d42b69efe1878c0d9bc19" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5543ddab5b484642b139b7c848932a58"
I20240502 21:29:48.688853  9105 raft_consensus.cc:3037] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:48.691776  9377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be34ec89876d42b69efe1878c0d9bc19" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
I20240502 21:29:48.692121  9377 raft_consensus.cc:3032] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:48.692226  9377 raft_consensus.cc:729] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Running, Role: LEADER
I20240502 21:29:48.692678  9377 consensus_queue.cc:260] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:48.693082  9377 raft_consensus.cc:3037] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:48.714583  9105 raft_consensus.cc:2454] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f162e164854f2da623a4bb8f9e6bf6 in term 2.
I20240502 21:29:48.715526  9377 raft_consensus.cc:2454] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f162e164854f2da623a4bb8f9e6bf6 in term 2.
I20240502 21:29:48.717769  9446 leader_election.cc:302] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 5543ddab5b484642b139b7c848932a58, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:29:48.718225 15937 raft_consensus.cc:2781] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:48.718358 15937 raft_consensus.cc:686] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 LEADER]: Becoming Leader. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Running, Role: LEADER
I20240502 21:29:48.718684 15937 consensus_queue.cc:237] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:48.717635  9507 leader_election.cc:288] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 2 election: Requested vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
I20240502 21:29:48.736758  9105 raft_consensus.cc:1226] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Rejecting Update request from peer f2fdd596a8b44bbc848fe40f9f07fbe0 for earlier term 1. Current term is 2. Ops: []
I20240502 21:29:48.747453 17176 ts_tablet_manager.cc:902] T a7c6b881c9ef4cdca787f065778e526f P d10e9e156ba74a97a221c85b68ab8bc4: Initiating tablet copy from peer e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467)
W20240502 21:29:48.747963  9173 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.69:42049, user_credentials={real_user=slave}} blocked reactor thread for 87428us
W20240502 21:29:48.750113  9313 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.72:40467, user_credentials={real_user=slave}} blocked reactor thread for 96594.9us
I20240502 21:29:48.750550  8962 catalog_manager.cc:5484] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: term changed from 1 to 2, leader changed from f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) to 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68). New cstate: current_term: 2 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:48.752722 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:48.771788  8949 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 51936fb1f678458a8795f8246590addb with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 21:29:48.776247 11093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be34ec89876d42b69efe1878c0d9bc19" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d10e9e156ba74a97a221c85b68ab8bc4"
I20240502 21:29:48.776575 11093 raft_consensus.cc:3037] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:48.779901 11093 raft_consensus.cc:2454] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f162e164854f2da623a4bb8f9e6bf6 in term 2.
I20240502 21:29:48.791728  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:48.792594 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:48.812906  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:48.844609 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:48.883608 17176 tablet_copy_client.cc:308] T a7c6b881c9ef4cdca787f065778e526f P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.72:40467
I20240502 21:29:48.884640 11517 tablet_copy_service.cc:140] P e0de0545fa0547d3929111e990e6d1b2: Received BeginTabletCopySession request for tablet a7c6b881c9ef4cdca787f065778e526f from peer d10e9e156ba74a97a221c85b68ab8bc4 ({username='slave'} at 127.27.196.69:43471)
I20240502 21:29:48.884822 11517 tablet_copy_service.cc:161] P e0de0545fa0547d3929111e990e6d1b2: Beginning new tablet copy session on tablet a7c6b881c9ef4cdca787f065778e526f from peer d10e9e156ba74a97a221c85b68ab8bc4 at {username='slave'} at 127.27.196.69:43471: session id = d10e9e156ba74a97a221c85b68ab8bc4-a7c6b881c9ef4cdca787f065778e526f
I20240502 21:29:48.888640 11517 tablet_copy_source_session.cc:205] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:48.895021 17176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7c6b881c9ef4cdca787f065778e526f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:48.919003 17176 tablet_copy_client.cc:783] T a7c6b881c9ef4cdca787f065778e526f P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:48.920115 17176 tablet_copy_client.cc:647] T a7c6b881c9ef4cdca787f065778e526f P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:48.923820 17176 tablet_copy_client.cc:523] T a7c6b881c9ef4cdca787f065778e526f P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:48.933015 17176 tablet_bootstrap.cc:492] T a7c6b881c9ef4cdca787f065778e526f P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap starting.
I20240502 21:29:48.964787  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:29:49.101269 17176 tablet_bootstrap.cc:492] T a7c6b881c9ef4cdca787f065778e526f P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:49.105288 17176 tablet_bootstrap.cc:492] T a7c6b881c9ef4cdca787f065778e526f P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap complete.
I20240502 21:29:49.105890 17176 ts_tablet_manager.cc:1367] T a7c6b881c9ef4cdca787f065778e526f P d10e9e156ba74a97a221c85b68ab8bc4: Time spent bootstrapping tablet: real 0.173s	user 0.083s	sys 0.013s
I20240502 21:29:49.107035 17176 raft_consensus.cc:348] T a7c6b881c9ef4cdca787f065778e526f P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:49.107491 17176 raft_consensus.cc:729] T a7c6b881c9ef4cdca787f065778e526f P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d10e9e156ba74a97a221c85b68ab8bc4, State: Initialized, Role: FOLLOWER
I20240502 21:29:49.108258 17176 consensus_queue.cc:260] T a7c6b881c9ef4cdca787f065778e526f P d10e9e156ba74a97a221c85b68ab8bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:49.109740 17176 ts_tablet_manager.cc:1398] T a7c6b881c9ef4cdca787f065778e526f P d10e9e156ba74a97a221c85b68ab8bc4: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 21:29:49.112617 11517 tablet_copy_service.cc:342] P e0de0545fa0547d3929111e990e6d1b2: Request end of tablet copy session d10e9e156ba74a97a221c85b68ab8bc4-a7c6b881c9ef4cdca787f065778e526f received from {username='slave'} at 127.27.196.69:43471
I20240502 21:29:49.112828 11517 tablet_copy_service.cc:433] P e0de0545fa0547d3929111e990e6d1b2: ending tablet copy session d10e9e156ba74a97a221c85b68ab8bc4-a7c6b881c9ef4cdca787f065778e526f on tablet a7c6b881c9ef4cdca787f065778e526f with peer d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:29:49.184317 17230 ts_tablet_manager.cc:902] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2: Initiating tablet copy from peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:29:49.194687 11093 raft_consensus.cc:1203] T a7c6b881c9ef4cdca787f065778e526f P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8]   Dedup: 2.8->[]
I20240502 21:29:49.294353  9377 raft_consensus.cc:1261] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:29:49.295445 15937 consensus_queue.cc:1035] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:49.314567 11093 raft_consensus.cc:1261] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:29:49.315418 15937 consensus_queue.cc:1035] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:49.320097  9105 raft_consensus.cc:1261] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:29:49.323915 15937 consensus_queue.cc:1035] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:49.381937  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:49.413311 17230 tablet_copy_client.cc:308] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:39071
I20240502 21:29:49.416046  9263 tablet_copy_service.cc:140] P 525a093ea5bf40dc9bbf9dc39895d670: Received BeginTabletCopySession request for tablet 51936fb1f678458a8795f8246590addb from peer e0de0545fa0547d3929111e990e6d1b2 ({username='slave'} at 127.27.196.72:35369)
I20240502 21:29:49.416256  9263 tablet_copy_service.cc:161] P 525a093ea5bf40dc9bbf9dc39895d670: Beginning new tablet copy session on tablet 51936fb1f678458a8795f8246590addb from peer e0de0545fa0547d3929111e990e6d1b2 at {username='slave'} at 127.27.196.72:35369: session id = e0de0545fa0547d3929111e990e6d1b2-51936fb1f678458a8795f8246590addb
I20240502 21:29:49.419713  9263 tablet_copy_source_session.cc:205] T 51936fb1f678458a8795f8246590addb P 525a093ea5bf40dc9bbf9dc39895d670: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:49.421777 17230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51936fb1f678458a8795f8246590addb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:49.448014 11209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:49.467475 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:49.517725  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:49.519448  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:49.529021 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:29:49.533793  9444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.65:42177, user_credentials={real_user=slave}} blocked reactor thread for 95249us
I20240502 21:29:49.542181 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:49.550261 17230 tablet_copy_client.cc:783] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:49.550779 17230 tablet_copy_client.cc:647] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:49.571281 17230 tablet_copy_client.cc:523] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:49.602394 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:49.605990 11209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:49.627938  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:29:49.631562  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:29:49.646028 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:49.654964 17230 tablet_bootstrap.cc:492] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2: Bootstrap starting.
I20240502 21:29:49.657369  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:49.704418  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:29:49.728029 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:49.789388  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:29:50.017488  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:50.029567 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:50.047838  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:29:50.071450 11209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:50.079195 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:50.083760 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:50.085258  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:50.134450  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:29:50.193130  9507 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f82c35077224414cb284472d49d7de4f"
dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34436
I20240502 21:29:50.193516  9507 raft_consensus.cc:595] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:29:50.205756  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:29:50.211367  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 21:29:50.215724 11028 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.66:39071, user_credentials={real_user=slave}} blocked reactor thread for 53681.1us
I20240502 21:29:50.231046 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:50.270056  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:50.332109  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:29:50.339509 17376 raft_consensus.cc:979] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6: : Instructing follower f2fdd596a8b44bbc848fe40f9f07fbe0 to start an election
I20240502 21:29:50.339819 17284 raft_consensus.cc:1067] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 LEADER]: Signalling peer f2fdd596a8b44bbc848fe40f9f07fbe0 to start an election
I20240502 21:29:50.340925  9378 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f82c35077224414cb284472d49d7de4f"
dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
 from {username='slave'} at 127.27.196.68:38747
I20240502 21:29:50.341181  9378 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:50.341310  9378 raft_consensus.cc:3037] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:50.345986  9378 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:50.355015  9507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f82c35077224414cb284472d49d7de4f" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6"
I20240502 21:29:50.355340  9507 raft_consensus.cc:3032] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:29:50.355423  9507 raft_consensus.cc:729] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Running, Role: LEADER
I20240502 21:29:50.355783  9507 consensus_queue.cc:260] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:50.356132  9239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f82c35077224414cb284472d49d7de4f" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670"
I20240502 21:29:50.356328  9507 raft_consensus.cc:3037] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:50.356534  9239 raft_consensus.cc:3037] T f82c35077224414cb284472d49d7de4f P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:50.361521  9507 raft_consensus.cc:2454] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 2.
I20240502 21:29:50.364311 11209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:29:50.375805 17230 log.cc:924] Time spent T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
I20240502 21:29:50.377655 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:50.384588  9378 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 2 election: Requested vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467)
I20240502 21:29:50.386837 11496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f82c35077224414cb284472d49d7de4f" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e0de0545fa0547d3929111e990e6d1b2"
I20240502 21:29:50.387451 11496 raft_consensus.cc:3037] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:50.390841 11496 raft_consensus.cc:2454] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 2.
I20240502 21:29:50.394322 11228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f82c35077224414cb284472d49d7de4f" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe"
I20240502 21:29:50.399595  9313 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, e0de0545fa0547d3929111e990e6d1b2, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:29:50.400462 17172 raft_consensus.cc:2781] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:29:50.401368 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:50.402328 11228 raft_consensus.cc:3037] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:29:50.406520 11228 raft_consensus.cc:2454] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 2.
I20240502 21:29:50.419000  9239 raft_consensus.cc:2454] T f82c35077224414cb284472d49d7de4f P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 2.
I20240502 21:29:50.434401 17172 raft_consensus.cc:686] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 LEADER]: Becoming Leader. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Running, Role: LEADER
I20240502 21:29:50.435267 17172 consensus_queue.cc:237] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:50.449009  8963 catalog_manager.cc:5484] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: term changed from 1 to 2, leader changed from 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) to f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67). New cstate: current_term: 2 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:50.545818 11228 consensus_queue.cc:237] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:50.572871  9105 raft_consensus.cc:1261] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Refusing update from remote peer 6ef0beaf2f6941afb6ea9f9f274517fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:29:50.574219 17373 consensus_queue.cc:1035] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:50.597175 17373 raft_consensus.cc:2932] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:50.603540  9105 raft_consensus.cc:2932] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:50.610538  8962 catalog_manager.cc:5484] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe reported cstate change: config changed from index 7 to 9, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New cstate: current_term: 2 leader_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:50.619256 17230 tablet_bootstrap.cc:492] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:50.620148 17230 tablet_bootstrap.cc:492] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2: Bootstrap complete.
I20240502 21:29:50.620514 17230 ts_tablet_manager.cc:1367] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2: Time spent bootstrapping tablet: real 0.966s	user 0.058s	sys 0.009s
I20240502 21:29:50.621506 17230 raft_consensus.cc:348] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:50.621826 17230 raft_consensus.cc:729] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0de0545fa0547d3929111e990e6d1b2, State: Initialized, Role: FOLLOWER
I20240502 21:29:50.622192 17230 consensus_queue.cc:260] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:50.623808 17230 ts_tablet_manager.cc:1398] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 21:29:50.624632  9263 tablet_copy_service.cc:342] P 525a093ea5bf40dc9bbf9dc39895d670: Request end of tablet copy session e0de0545fa0547d3929111e990e6d1b2-51936fb1f678458a8795f8246590addb received from {username='slave'} at 127.27.196.72:35369
I20240502 21:29:50.624779  9263 tablet_copy_service.cc:433] P 525a093ea5bf40dc9bbf9dc39895d670: ending tablet copy session e0de0545fa0547d3929111e990e6d1b2-51936fb1f678458a8795f8246590addb on tablet 51936fb1f678458a8795f8246590addb with peer e0de0545fa0547d3929111e990e6d1b2
I20240502 21:29:50.630416  9221 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d1a31513bb64578a85adf1f622e5bc0 with delete_type TABLET_DATA_TOMBSTONED (TS 525a093ea5bf40dc9bbf9dc39895d670 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:44408
I20240502 21:29:50.631610 11228 consensus_queue.cc:237] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:50.638682  9105 raft_consensus.cc:1261] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Refusing update from remote peer 6ef0beaf2f6941afb6ea9f9f274517fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 21:29:50.641415 17001 consensus_queue.cc:1035] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:50.645058 17001 raft_consensus.cc:2932] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
W20240502 21:29:50.647871 11163 consensus_peers.cc:485] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe -> Peer e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467): Couldn't send request to peer e0de0545fa0547d3929111e990e6d1b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4d1a31513bb64578a85adf1f622e5bc0. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:50.650843  9107 raft_consensus.cc:2932] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:50.657959  8963 catalog_manager.cc:5484] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index 9 to 10, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New cstate: current_term: 2 leader_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:50.672463  8946 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 4d1a31513bb64578a85adf1f622e5bc0 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 21:29:50.705742 17395 tablet_replica.cc:330] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670: stopping tablet replica
I20240502 21:29:50.708874 17395 raft_consensus.cc:2229] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:50.709504 17395 raft_consensus.cc:2258] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:50.715013 17395 ts_tablet_manager.cc:1875] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:50.773124 17395 ts_tablet_manager.cc:1888] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240502 21:29:50.784575 17395 log.cc:1196] T 4d1a31513bb64578a85adf1f622e5bc0 P 525a093ea5bf40dc9bbf9dc39895d670: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/wals/4d1a31513bb64578a85adf1f622e5bc0
I20240502 21:29:50.795519  8949 catalog_manager.cc:4830] TS 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): tablet 4d1a31513bb64578a85adf1f622e5bc0 (table rebalance_test_table_1 [id=f52c6c3804274777abfa00bb33ae9b16]) successfully deleted
I20240502 21:29:50.831022 11496 raft_consensus.cc:1261] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:29:50.833283 17172 consensus_queue.cc:1035] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:50.838433 11228 raft_consensus.cc:1261] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:29:50.839689 17172 consensus_queue.cc:1035] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:50.934584  9508 raft_consensus.cc:1261] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:29:50.935680 17172 consensus_queue.cc:1035] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:50.962497  9239 raft_consensus.cc:1261] T f82c35077224414cb284472d49d7de4f P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:29:50.963893 17380 consensus_queue.cc:1035] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:51.035863 11496 raft_consensus.cc:1203] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
W20240502 21:29:51.049358  9445 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.69:42049, user_credentials={real_user=slave}} blocked reactor thread for 102112us
I20240502 21:29:51.103200 17230 ts_tablet_manager.cc:902] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2: Initiating tablet copy from peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211)
I20240502 21:29:51.114135  9105 consensus_queue.cc:237] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:51.142309 11228 raft_consensus.cc:1261] T 32c306d724954a16be21a34aa4cc0b5f P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:51.142472 11364 raft_consensus.cc:1261] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:51.143129 17290 consensus_queue.cc:1035] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:51.145154 17290 consensus_queue.cc:1035] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:51.177124 17290 raft_consensus.cc:2932] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:51.181030 11229 raft_consensus.cc:2932] T 32c306d724954a16be21a34aa4cc0b5f P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:51.197052 11365 raft_consensus.cc:2932] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:51.200717  8960 catalog_manager.cc:5484] T 32c306d724954a16be21a34aa4cc0b5f P 6ef0beaf2f6941afb6ea9f9f274517fe reported cstate change: config changed from index 6 to 7, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:51.274595 17230 tablet_copy_client.cc:308] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.70:44211
I20240502 21:29:51.276309 11250 tablet_copy_service.cc:140] P 6ef0beaf2f6941afb6ea9f9f274517fe: Received BeginTabletCopySession request for tablet 4d1a31513bb64578a85adf1f622e5bc0 from peer e0de0545fa0547d3929111e990e6d1b2 ({username='slave'} at 127.27.196.72:41013)
I20240502 21:29:51.276608 11250 tablet_copy_service.cc:161] P 6ef0beaf2f6941afb6ea9f9f274517fe: Beginning new tablet copy session on tablet 4d1a31513bb64578a85adf1f622e5bc0 from peer e0de0545fa0547d3929111e990e6d1b2 at {username='slave'} at 127.27.196.72:41013: session id = e0de0545fa0547d3929111e990e6d1b2-4d1a31513bb64578a85adf1f622e5bc0
I20240502 21:29:51.285394 11250 tablet_copy_source_session.cc:205] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:51.288548 17230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d1a31513bb64578a85adf1f622e5bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:51.294898  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:51.301519 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:51.359509  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:51.377543  9221 tablet_service.cc:1514] Processing DeleteTablet for tablet 32c306d724954a16be21a34aa4cc0b5f with delete_type TABLET_DATA_TOMBSTONED (TS 525a093ea5bf40dc9bbf9dc39895d670 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44408
I20240502 21:29:51.408521 17525 tablet_replica.cc:330] T 32c306d724954a16be21a34aa4cc0b5f P 525a093ea5bf40dc9bbf9dc39895d670: stopping tablet replica
I20240502 21:29:51.409052 17525 raft_consensus.cc:2229] T 32c306d724954a16be21a34aa4cc0b5f P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:51.409691 17525 raft_consensus.cc:2258] T 32c306d724954a16be21a34aa4cc0b5f P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:51.410959 17525 ts_tablet_manager.cc:1875] T 32c306d724954a16be21a34aa4cc0b5f P 525a093ea5bf40dc9bbf9dc39895d670: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:51.425379  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:51.430037 11209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:51.436676 17230 tablet_copy_client.cc:783] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:51.445472 17230 tablet_copy_client.cc:647] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:51.449959 17230 tablet_copy_client.cc:523] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:51.455649 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:51.465081 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:51.496469 17230 tablet_bootstrap.cc:492] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2: Bootstrap starting.
I20240502 21:29:51.636071  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:51.684845 17525 ts_tablet_manager.cc:1888] T 32c306d724954a16be21a34aa4cc0b5f P 525a093ea5bf40dc9bbf9dc39895d670: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:29:51.685209 17525 log.cc:1196] T 32c306d724954a16be21a34aa4cc0b5f P 525a093ea5bf40dc9bbf9dc39895d670: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/wals/32c306d724954a16be21a34aa4cc0b5f
I20240502 21:29:51.686444  8949 catalog_manager.cc:4830] TS 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): tablet 32c306d724954a16be21a34aa4cc0b5f (table rebalance_test_table_5 [id=e8eef05f850243d0bdc7e1899f8c6139]) successfully deleted
I20240502 21:29:51.699731  9507 consensus_queue.cc:237] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:51.721798 17230 tablet_bootstrap.cc:492] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:51.722347 17230 tablet_bootstrap.cc:492] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2: Bootstrap complete.
I20240502 21:29:51.722755 17230 ts_tablet_manager.cc:1367] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2: Time spent bootstrapping tablet: real 0.226s	user 0.068s	sys 0.010s
I20240502 21:29:51.723729 17230 raft_consensus.cc:348] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:51.724007 17230 raft_consensus.cc:729] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0de0545fa0547d3929111e990e6d1b2, State: Initialized, Role: FOLLOWER
I20240502 21:29:51.724308 17230 consensus_queue.cc:260] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:51.725535 11093 raft_consensus.cc:1261] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:29:51.725895 17230 ts_tablet_manager.cc:1398] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2: Time spent starting tablet: real 0.003s	user 0.005s	sys 0.000s
I20240502 21:29:51.726912 11250 tablet_copy_service.cc:342] P 6ef0beaf2f6941afb6ea9f9f274517fe: Request end of tablet copy session e0de0545fa0547d3929111e990e6d1b2-4d1a31513bb64578a85adf1f622e5bc0 received from {username='slave'} at 127.27.196.72:41013
I20240502 21:29:51.727079 11250 tablet_copy_service.cc:433] P 6ef0beaf2f6941afb6ea9f9f274517fe: ending tablet copy session e0de0545fa0547d3929111e990e6d1b2-4d1a31513bb64578a85adf1f622e5bc0 on tablet 4d1a31513bb64578a85adf1f622e5bc0 with peer e0de0545fa0547d3929111e990e6d1b2
I20240502 21:29:51.729265 17451 consensus_queue.cc:1035] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:51.737066  9105 raft_consensus.cc:1261] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:29:51.739717 17451 consensus_queue.cc:1035] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:51.800981 17376 raft_consensus.cc:2932] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:51.817981 11092 raft_consensus.cc:2932] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:51.827414  9105 raft_consensus.cc:2932] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:51.829049  8963 catalog_manager.cc:5484] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: config changed from index 4 to 6, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New cstate: current_term: 2 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:51.847988  9356 tablet_service.cc:1514] Processing DeleteTablet for tablet be34ec89876d42b69efe1878c0d9bc19 with delete_type TABLET_DATA_TOMBSTONED (TS f2fdd596a8b44bbc848fe40f9f07fbe0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55106
I20240502 21:29:51.870772 17568 tablet_replica.cc:330] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0: stopping tablet replica
I20240502 21:29:51.871915 17568 raft_consensus.cc:2229] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:51.880185 17568 raft_consensus.cc:2258] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:51.906497 17568 ts_tablet_manager.cc:1875] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:51.926080 17568 ts_tablet_manager.cc:1888] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 21:29:51.926268 17568 log.cc:1196] T be34ec89876d42b69efe1878c0d9bc19 P f2fdd596a8b44bbc848fe40f9f07fbe0: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/wals/be34ec89876d42b69efe1878c0d9bc19
I20240502 21:29:51.940255  8948 catalog_manager.cc:4830] TS f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): tablet be34ec89876d42b69efe1878c0d9bc19 (table rebalance_test_table_6 [id=265d5656d7c048aa9795a906661b8ea8]) successfully deleted
I20240502 21:29:52.128957  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:52.186904  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:52.227883 11495 raft_consensus.cc:1203] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10]   Dedup: 2.10->[]
I20240502 21:29:52.230283 11209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:52.250211 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:52.251622 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:52.296166  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:52.304793  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:52.311484 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:52.570091  9378 consensus_queue.cc:237] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:52.573526 11495 raft_consensus.cc:1261] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:29:52.574457  9239 raft_consensus.cc:1261] T f82c35077224414cb284472d49d7de4f P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:29:52.576004 11229 raft_consensus.cc:1261] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:29:52.576480 17671 consensus_queue.cc:1035] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:52.577461 17455 consensus_queue.cc:1035] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:52.578557 17455 consensus_queue.cc:1035] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:52.592432 17671 raft_consensus.cc:2932] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:52.593168  9239 raft_consensus.cc:2932] T f82c35077224414cb284472d49d7de4f P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:52.599928  8962 catalog_manager.cc:5484] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: config changed from index 5 to 7, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New cstate: current_term: 2 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:52.597965 11495 raft_consensus.cc:2932] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:52.633004 11229 raft_consensus.cc:2932] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:52.639375  9490 tablet_service.cc:1514] Processing DeleteTablet for tablet f82c35077224414cb284472d49d7de4f with delete_type TABLET_DATA_TOMBSTONED (TS 19f162e164854f2da623a4bb8f9e6bf6 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54670
I20240502 21:29:52.642503 17684 tablet_replica.cc:330] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6: stopping tablet replica
I20240502 21:29:52.643750 17684 raft_consensus.cc:2229] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:52.644127 17684 raft_consensus.cc:2258] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:52.645057 17684 ts_tablet_manager.cc:1875] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:52.657704 17684 ts_tablet_manager.cc:1888] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 21:29:52.658134 17684 log.cc:1196] T f82c35077224414cb284472d49d7de4f P 19f162e164854f2da623a4bb8f9e6bf6: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/wals/f82c35077224414cb284472d49d7de4f
I20240502 21:29:52.659711  8945 catalog_manager.cc:4830] TS 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): tablet f82c35077224414cb284472d49d7de4f (table rebalance_test_table_5 [id=e8eef05f850243d0bdc7e1899f8c6139]) successfully deleted
I20240502 21:29:52.814306  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:52.887814 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:52.888393  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:52.936223  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:53.020254 11209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:53.053570 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:53.074992 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:53.099542  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:53.134166  9105 consensus_queue.cc:237] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:53.155417 11092 raft_consensus.cc:1261] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:53.155512  9507 raft_consensus.cc:1261] T de3287296c4547688388388dd4c7e638 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:53.157176 17561 consensus_queue.cc:1035] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:53.159875 17562 consensus_queue.cc:1035] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:53.165117 17561 raft_consensus.cc:2932] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:53.168766 11092 raft_consensus.cc:2932] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:53.199352  8962 catalog_manager.cc:5484] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4 reported cstate change: config changed from index 4 to 5, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:53.225035  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:53.229990  9507 raft_consensus.cc:2932] T de3287296c4547688388388dd4c7e638 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:53.280428  9356 tablet_service.cc:1514] Processing DeleteTablet for tablet de3287296c4547688388388dd4c7e638 with delete_type TABLET_DATA_TOMBSTONED (TS f2fdd596a8b44bbc848fe40f9f07fbe0 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55106
I20240502 21:29:53.281592 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:53.282141 17786 tablet_replica.cc:330] T de3287296c4547688388388dd4c7e638 P f2fdd596a8b44bbc848fe40f9f07fbe0: stopping tablet replica
I20240502 21:29:53.282423 17786 raft_consensus.cc:2229] T de3287296c4547688388388dd4c7e638 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:53.282756 17786 raft_consensus.cc:2258] T de3287296c4547688388388dd4c7e638 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:53.284047 17786 ts_tablet_manager.cc:1875] T de3287296c4547688388388dd4c7e638 P f2fdd596a8b44bbc848fe40f9f07fbe0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:53.292232  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:53.315313 11209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:53.325387  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:53.335325 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:53.361872  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:53.371656 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:53.499437 17786 ts_tablet_manager.cc:1888] T de3287296c4547688388388dd4c7e638 P f2fdd596a8b44bbc848fe40f9f07fbe0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:29:53.499732 17786 log.cc:1196] T de3287296c4547688388388dd4c7e638 P f2fdd596a8b44bbc848fe40f9f07fbe0: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/wals/de3287296c4547688388388dd4c7e638
I20240502 21:29:53.500701  8948 catalog_manager.cc:4830] TS f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): tablet de3287296c4547688388388dd4c7e638 (table rebalance_test_table_0 [id=445d2e6ec7154b80abec5fff346b8f5e]) successfully deleted
I20240502 21:29:53.519521  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:53.593730  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:53.601269 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:53.622212 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:53.655820  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:53.656634  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:53.656759 11209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:53.693280 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:53.820672  9105 consensus_queue.cc:237] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:53.837014 11364 raft_consensus.cc:1261] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:53.837044  9239 raft_consensus.cc:1261] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:53.838140 17562 consensus_queue.cc:1035] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:53.841956 17562 consensus_queue.cc:1035] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:53.844965 17768 raft_consensus.cc:2932] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:53.851585 11364 raft_consensus.cc:2932] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:53.862110 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:53.875627  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:53.885278  8961 catalog_manager.cc:5484] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index 4 to 5, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:53.889533  9239 raft_consensus.cc:2932] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:53.900056  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 21:29:53.931435  9038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.71:35857, user_credentials={real_user=slave}} blocked reactor thread for 51679.1us
I20240502 21:29:53.976888  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:54.016448  9490 tablet_service.cc:1514] Processing DeleteTablet for tablet cdf72d25f0f847e5998acbc53979e64d with delete_type TABLET_DATA_TOMBSTONED (TS 19f162e164854f2da623a4bb8f9e6bf6 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54670
I20240502 21:29:54.030869 17885 tablet_replica.cc:330] T cdf72d25f0f847e5998acbc53979e64d P 19f162e164854f2da623a4bb8f9e6bf6: stopping tablet replica
I20240502 21:29:54.031497 11209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:54.034511 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:54.034855 17885 raft_consensus.cc:2229] T cdf72d25f0f847e5998acbc53979e64d P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:54.036515 17885 raft_consensus.cc:2258] T cdf72d25f0f847e5998acbc53979e64d P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:54.038182 17885 ts_tablet_manager.cc:1875] T cdf72d25f0f847e5998acbc53979e64d P 19f162e164854f2da623a4bb8f9e6bf6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:54.068390 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:54.129201 17885 ts_tablet_manager.cc:1888] T cdf72d25f0f847e5998acbc53979e64d P 19f162e164854f2da623a4bb8f9e6bf6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:29:54.129419 17885 log.cc:1196] T cdf72d25f0f847e5998acbc53979e64d P 19f162e164854f2da623a4bb8f9e6bf6: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/wals/cdf72d25f0f847e5998acbc53979e64d
I20240502 21:29:54.132292  8945 catalog_manager.cc:4830] TS 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): tablet cdf72d25f0f847e5998acbc53979e64d (table rebalance_test_table_2 [id=c0a0ef39a2a246449eb582cc1d993484]) successfully deleted
I20240502 21:29:54.142822  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:54.186532  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:54.213821 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:54.239475 11209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:54.263701 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:54.265192 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:54.272007  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:54.302350  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:54.314621  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:54.413654 11229 consensus_queue.cc:237] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:54.442914 11364 raft_consensus.cc:1261] T 07d693b680a54e12871121737e3da744 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Refusing update from remote peer 6ef0beaf2f6941afb6ea9f9f274517fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:29:54.443284 11092 raft_consensus.cc:1261] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Refusing update from remote peer 6ef0beaf2f6941afb6ea9f9f274517fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:29:54.443881 17571 consensus_queue.cc:1035] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:54.444682 17571 consensus_queue.cc:1035] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:54.461475  9507 raft_consensus.cc:1261] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Refusing update from remote peer 6ef0beaf2f6941afb6ea9f9f274517fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:29:54.464056 17571 consensus_queue.cc:1035] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:54.491663  9378 consensus_queue.cc:237] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:54.513845 17571 raft_consensus.cc:2932] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:54.514603 11092 raft_consensus.cc:1261] T 8a2e06253c51405fa9b6bc9e87844760 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 21:29:54.515494 11160 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.68:40931, user_credentials={real_user=slave}} blocked reactor thread for 53251.7us
I20240502 21:29:54.516500 17172 consensus_queue.cc:1035] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:54.517412 11092 raft_consensus.cc:2932] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:54.530737 11364 raft_consensus.cc:2932] T 07d693b680a54e12871121737e3da744 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:54.537223  8953 catalog_manager.cc:5484] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe reported cstate change: config changed from index 8 to 9, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New cstate: current_term: 2 leader_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
W20240502 21:29:54.576225  9310 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.71:35857, user_credentials={real_user=slave}} blocked reactor thread for 60305.1us
I20240502 21:29:54.576074 17924 raft_consensus.cc:2932] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:54.578881 11093 raft_consensus.cc:2932] T 8a2e06253c51405fa9b6bc9e87844760 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:54.578930  9507 raft_consensus.cc:2932] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:54.621486  9356 tablet_service.cc:1514] Processing DeleteTablet for tablet 07d693b680a54e12871121737e3da744 with delete_type TABLET_DATA_TOMBSTONED (TS f2fdd596a8b44bbc848fe40f9f07fbe0 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55106
I20240502 21:29:54.622805 17965 tablet_replica.cc:330] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0: stopping tablet replica
I20240502 21:29:54.623270 17965 raft_consensus.cc:2229] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:54.623917  8960 catalog_manager.cc:5484] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: config changed from index 5 to 6, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:54.625113 17965 raft_consensus.cc:2258] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:54.633987  9490 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a2e06253c51405fa9b6bc9e87844760 with delete_type TABLET_DATA_TOMBSTONED (TS 19f162e164854f2da623a4bb8f9e6bf6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54670
I20240502 21:29:54.634017  9378 consensus_queue.cc:237] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:54.652251 11093 raft_consensus.cc:1261] T 8a2e06253c51405fa9b6bc9e87844760 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:54.653156 17172 consensus_queue.cc:1035] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 21:29:54.656399  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a2e06253c51405fa9b6bc9e87844760. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:54.659348 11093 raft_consensus.cc:2932] T 8a2e06253c51405fa9b6bc9e87844760 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:54.663964 17969 tablet_replica.cc:330] T 8a2e06253c51405fa9b6bc9e87844760 P 19f162e164854f2da623a4bb8f9e6bf6: stopping tablet replica
I20240502 21:29:54.663457 17958 raft_consensus.cc:2932] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:54.667716 17969 raft_consensus.cc:2229] T 8a2e06253c51405fa9b6bc9e87844760 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:54.673164  8959 catalog_manager.cc:5484] T 8a2e06253c51405fa9b6bc9e87844760 P d10e9e156ba74a97a221c85b68ab8bc4 reported cstate change: config changed from index 6 to 7, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:54.633062 17965 ts_tablet_manager.cc:1875] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:54.683521 17969 raft_consensus.cc:2258] T 8a2e06253c51405fa9b6bc9e87844760 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:54.686723 17969 ts_tablet_manager.cc:1875] T 8a2e06253c51405fa9b6bc9e87844760 P 19f162e164854f2da623a4bb8f9e6bf6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:54.696580  8948 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 8a2e06253c51405fa9b6bc9e87844760 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 21:29:54.722438 17969 ts_tablet_manager.cc:1888] T 8a2e06253c51405fa9b6bc9e87844760 P 19f162e164854f2da623a4bb8f9e6bf6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:29:54.731403 17969 log.cc:1196] T 8a2e06253c51405fa9b6bc9e87844760 P 19f162e164854f2da623a4bb8f9e6bf6: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/wals/8a2e06253c51405fa9b6bc9e87844760
I20240502 21:29:54.736137  8945 catalog_manager.cc:4830] TS 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): tablet 8a2e06253c51405fa9b6bc9e87844760 (table rebalance_test_table_5 [id=e8eef05f850243d0bdc7e1899f8c6139]) successfully deleted
I20240502 21:29:54.746878 17965 ts_tablet_manager.cc:1888] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240502 21:29:54.747069 17965 log.cc:1196] T 07d693b680a54e12871121737e3da744 P f2fdd596a8b44bbc848fe40f9f07fbe0: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/wals/07d693b680a54e12871121737e3da744
I20240502 21:29:54.760038  8948 catalog_manager.cc:4830] TS f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): tablet 07d693b680a54e12871121737e3da744 (table rebalance_test_table_1 [id=f52c6c3804274777abfa00bb33ae9b16]) successfully deleted
I20240502 21:29:54.869251  9239 consensus_queue.cc:237] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:54.884007 11229 raft_consensus.cc:1261] T 3caa36212ba444aa88d153ff9f90122e P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:54.884828 11093 raft_consensus.cc:1261] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:54.885630  9378 raft_consensus.cc:1261] T 3caa36212ba444aa88d153ff9f90122e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:54.886626 17950 consensus_queue.cc:1035] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:54.891202 17950 consensus_queue.cc:1035] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:54.892058 17950 consensus_queue.cc:1035] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:54.917723 17950 raft_consensus.cc:2932] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:54.921594 11093 raft_consensus.cc:2932] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:54.926618 11229 raft_consensus.cc:2932] T 3caa36212ba444aa88d153ff9f90122e P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:54.945415  9378 raft_consensus.cc:2932] T 3caa36212ba444aa88d153ff9f90122e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:54.954223  8959 catalog_manager.cc:5484] T 3caa36212ba444aa88d153ff9f90122e P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: config changed from index 5 to 6, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:54.960597  9490 tablet_service.cc:1514] Processing DeleteTablet for tablet 3caa36212ba444aa88d153ff9f90122e with delete_type TABLET_DATA_TOMBSTONED (TS 19f162e164854f2da623a4bb8f9e6bf6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54670
I20240502 21:29:54.961221 17969 tablet_replica.cc:330] T 3caa36212ba444aa88d153ff9f90122e P 19f162e164854f2da623a4bb8f9e6bf6: stopping tablet replica
I20240502 21:29:54.961501 17969 raft_consensus.cc:2229] T 3caa36212ba444aa88d153ff9f90122e P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:54.961845 17969 raft_consensus.cc:2258] T 3caa36212ba444aa88d153ff9f90122e P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:54.962980 17969 ts_tablet_manager.cc:1875] T 3caa36212ba444aa88d153ff9f90122e P 19f162e164854f2da623a4bb8f9e6bf6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:54.979223 17969 ts_tablet_manager.cc:1888] T 3caa36212ba444aa88d153ff9f90122e P 19f162e164854f2da623a4bb8f9e6bf6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:29:54.979420 17969 log.cc:1196] T 3caa36212ba444aa88d153ff9f90122e P 19f162e164854f2da623a4bb8f9e6bf6: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/wals/3caa36212ba444aa88d153ff9f90122e
I20240502 21:29:54.980481  8945 catalog_manager.cc:4830] TS 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): tablet 3caa36212ba444aa88d153ff9f90122e (table rebalance_test_table_9 [id=16e8100140034581b2a773dcf740998d]) successfully deleted
I20240502 21:29:55.011965  9105 consensus_queue.cc:237] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:55.034951  9508 raft_consensus.cc:1261] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:55.034951 11229 raft_consensus.cc:1261] T dc1970b9263d41abb575b9d8639c59ae P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:55.035856 17768 consensus_queue.cc:1035] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:55.036768 17864 consensus_queue.cc:1035] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:55.050633 17864 raft_consensus.cc:2932] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:55.056550  9507 raft_consensus.cc:2932] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:55.064791  8959 catalog_manager.cc:5484] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index 4 to 5, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:55.072754 11228 raft_consensus.cc:2932] T dc1970b9263d41abb575b9d8639c59ae P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:55.086012  9221 tablet_service.cc:1514] Processing DeleteTablet for tablet dc1970b9263d41abb575b9d8639c59ae with delete_type TABLET_DATA_TOMBSTONED (TS 525a093ea5bf40dc9bbf9dc39895d670 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44408
I20240502 21:29:55.092470 18056 tablet_replica.cc:330] T dc1970b9263d41abb575b9d8639c59ae P 525a093ea5bf40dc9bbf9dc39895d670: stopping tablet replica
I20240502 21:29:55.093211 18056 raft_consensus.cc:2229] T dc1970b9263d41abb575b9d8639c59ae P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:55.094017 18056 raft_consensus.cc:2258] T dc1970b9263d41abb575b9d8639c59ae P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:55.095486 18056 ts_tablet_manager.cc:1875] T dc1970b9263d41abb575b9d8639c59ae P 525a093ea5bf40dc9bbf9dc39895d670: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:55.142354 18051 ts_tablet_manager.cc:902] T 8a2e06253c51405fa9b6bc9e87844760 P 6ef0beaf2f6941afb6ea9f9f274517fe: Initiating tablet copy from peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:29:55.167814 18051 tablet_copy_client.cc:308] T 8a2e06253c51405fa9b6bc9e87844760 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:43203
I20240502 21:29:55.169059  9399 tablet_copy_service.cc:140] P f2fdd596a8b44bbc848fe40f9f07fbe0: Received BeginTabletCopySession request for tablet 8a2e06253c51405fa9b6bc9e87844760 from peer 6ef0beaf2f6941afb6ea9f9f274517fe ({username='slave'} at 127.27.196.70:34655)
I20240502 21:29:55.169323  9399 tablet_copy_service.cc:161] P f2fdd596a8b44bbc848fe40f9f07fbe0: Beginning new tablet copy session on tablet 8a2e06253c51405fa9b6bc9e87844760 from peer 6ef0beaf2f6941afb6ea9f9f274517fe at {username='slave'} at 127.27.196.70:34655: session id = 6ef0beaf2f6941afb6ea9f9f274517fe-8a2e06253c51405fa9b6bc9e87844760
I20240502 21:29:55.171252  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:55.173903  9399 tablet_copy_source_session.cc:205] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:55.176661 18051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2e06253c51405fa9b6bc9e87844760. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:55.202988  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:55.238723 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:55.242125 18051 tablet_copy_client.cc:783] T 8a2e06253c51405fa9b6bc9e87844760 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:55.242470 18051 tablet_copy_client.cc:647] T 8a2e06253c51405fa9b6bc9e87844760 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:55.252727 18051 tablet_copy_client.cc:523] T 8a2e06253c51405fa9b6bc9e87844760 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:55.260504 18056 ts_tablet_manager.cc:1888] T dc1970b9263d41abb575b9d8639c59ae P 525a093ea5bf40dc9bbf9dc39895d670: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:29:55.265126 18056 log.cc:1196] T dc1970b9263d41abb575b9d8639c59ae P 525a093ea5bf40dc9bbf9dc39895d670: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/wals/dc1970b9263d41abb575b9d8639c59ae
I20240502 21:29:55.270681  8949 catalog_manager.cc:4830] TS 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): tablet dc1970b9263d41abb575b9d8639c59ae (table rebalance_test_table_4 [id=1b17408dbab9492bba0ed50efd6ef43f]) successfully deleted
I20240502 21:29:55.278085  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:55.319437 18051 tablet_bootstrap.cc:492] T 8a2e06253c51405fa9b6bc9e87844760 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap starting.
I20240502 21:29:55.328320 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:55.335840 11209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:55.342355 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:55.390043  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:55.584048 18051 tablet_bootstrap.cc:492] T 8a2e06253c51405fa9b6bc9e87844760 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:55.584559 18051 tablet_bootstrap.cc:492] T 8a2e06253c51405fa9b6bc9e87844760 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap complete.
I20240502 21:29:55.584869 18051 ts_tablet_manager.cc:1367] T 8a2e06253c51405fa9b6bc9e87844760 P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent bootstrapping tablet: real 0.266s	user 0.062s	sys 0.002s
I20240502 21:29:55.585749 18051 raft_consensus.cc:348] T 8a2e06253c51405fa9b6bc9e87844760 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:55.587378 18051 raft_consensus.cc:729] T 8a2e06253c51405fa9b6bc9e87844760 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef0beaf2f6941afb6ea9f9f274517fe, State: Initialized, Role: FOLLOWER
I20240502 21:29:55.588033 18051 consensus_queue.cc:260] T 8a2e06253c51405fa9b6bc9e87844760 P 6ef0beaf2f6941afb6ea9f9f274517fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:55.589546 18051 ts_tablet_manager.cc:1398] T 8a2e06253c51405fa9b6bc9e87844760 P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.002s
I20240502 21:29:55.594362  9399 tablet_copy_service.cc:342] P f2fdd596a8b44bbc848fe40f9f07fbe0: Request end of tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-8a2e06253c51405fa9b6bc9e87844760 received from {username='slave'} at 127.27.196.70:34655
I20240502 21:29:55.594532  9399 tablet_copy_service.cc:433] P f2fdd596a8b44bbc848fe40f9f07fbe0: ending tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-8a2e06253c51405fa9b6bc9e87844760 on tablet 8a2e06253c51405fa9b6bc9e87844760 with peer 6ef0beaf2f6941afb6ea9f9f274517fe
I20240502 21:29:55.681897 11229 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "07d693b680a54e12871121737e3da744"
dest_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:55592
I20240502 21:29:55.682246 11229 raft_consensus.cc:595] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 LEADER]: Received request to transfer leadership
I20240502 21:29:55.993671 11229 raft_consensus.cc:1203] T 8a2e06253c51405fa9b6bc9e87844760 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 21:29:56.012835  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:56.013932  9490 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:56.035270 11209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:56.106120 18210 raft_consensus.cc:979] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe: : Instructing follower 19f162e164854f2da623a4bb8f9e6bf6 to start an election
I20240502 21:29:56.106379 18210 raft_consensus.cc:1067] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 LEADER]: Signalling peer 19f162e164854f2da623a4bb8f9e6bf6 to start an election
I20240502 21:29:56.110320  9507 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "07d693b680a54e12871121737e3da744"
dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6"
 from {username='slave'} at 127.27.196.70:53371
I20240502 21:29:56.110822  9507 raft_consensus.cc:482] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:56.111023  9507 raft_consensus.cc:3037] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:56.132647  9507 raft_consensus.cc:504] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:56.162889 11229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d693b680a54e12871121737e3da744" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe"
I20240502 21:29:56.163201 11229 raft_consensus.cc:3032] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:56.163287 11229 raft_consensus.cc:729] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6ef0beaf2f6941afb6ea9f9f274517fe, State: Running, Role: LEADER
I20240502 21:29:56.163686 11229 consensus_queue.cc:260] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:56.164222 11229 raft_consensus.cc:3037] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:56.176076 11093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d693b680a54e12871121737e3da744" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "d10e9e156ba74a97a221c85b68ab8bc4"
I20240502 21:29:56.176363 11093 raft_consensus.cc:3037] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:56.182370 11093 raft_consensus.cc:2454] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f162e164854f2da623a4bb8f9e6bf6 in term 3.
I20240502 21:29:56.190790 11229 raft_consensus.cc:2454] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f162e164854f2da623a4bb8f9e6bf6 in term 3.
I20240502 21:29:56.191856  9445 leader_election.cc:302] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4; no voters: 
I20240502 21:29:56.192417 18005 raft_consensus.cc:2781] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:56.192999 18005 raft_consensus.cc:686] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [term 3 LEADER]: Becoming Leader. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Running, Role: LEADER
I20240502 21:29:56.193524 18005 consensus_queue.cc:237] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:56.199965  8959 catalog_manager.cc:5484] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: term changed from 2 to 3, leader changed from 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) to 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68). New cstate: current_term: 3 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:56.208065  9507 leader_election.cc:288] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 3 election: Requested vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049), 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857)
I20240502 21:29:56.226153 11364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d693b680a54e12871121737e3da744" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2"
I20240502 21:29:56.226403 11364 raft_consensus.cc:3037] T 07d693b680a54e12871121737e3da744 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:56.230530 11364 raft_consensus.cc:2454] T 07d693b680a54e12871121737e3da744 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f162e164854f2da623a4bb8f9e6bf6 in term 3.
I20240502 21:29:56.270915 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:56.303200 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:56.303866  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:56.401247 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:56.402248  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:56.587095  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:56.588547  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:29:56.630052  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:56.647612 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:56.664630 11209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:56.669929 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:56.716562 11229 raft_consensus.cc:1261] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 3 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 21:29:56.718422 18224 consensus_queue.cc:1035] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:56.781685  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:56.796486 11093 raft_consensus.cc:1261] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4 [term 3 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 21:29:56.799923 18005 consensus_queue.cc:1035] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:56.815837 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:56.818656 11364 raft_consensus.cc:1261] T 07d693b680a54e12871121737e3da744 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 3 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 21:29:56.830761 18005 consensus_queue.cc:1035] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:57.033210  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 21:29:57.068904  9311 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 48931.2us
I20240502 21:29:57.085213 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:57.192478 11209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:57.193243 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:57.194213  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:57.224731  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:57.250819  9105 consensus_queue.cc:237] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:57.286839  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:29:57.315423 11229 raft_consensus.cc:1261] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:57.317054 18050 consensus_queue.cc:1035] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:57.322654  9507 raft_consensus.cc:1261] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:57.323634 17864 consensus_queue.cc:1035] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:57.333014 18050 raft_consensus.cc:2932] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:57.359616 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:57.467599 11228 raft_consensus.cc:2932] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:57.483433  8962 catalog_manager.cc:5484] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 6ef0beaf2f6941afb6ea9f9f274517fe reported cstate change: config changed from index 4 to 5, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:57.507151  9507 raft_consensus.cc:2932] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:57.639295  9221 tablet_service.cc:1514] Processing DeleteTablet for tablet 9240163cfac446d2a14dbd9fcbbf9ff7 with delete_type TABLET_DATA_TOMBSTONED (TS 525a093ea5bf40dc9bbf9dc39895d670 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44408
I20240502 21:29:57.652191 18394 tablet_replica.cc:330] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 525a093ea5bf40dc9bbf9dc39895d670: stopping tablet replica
I20240502 21:29:57.652717 18394 raft_consensus.cc:2229] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:57.653012 18394 raft_consensus.cc:2258] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:57.653972 18394 ts_tablet_manager.cc:1875] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 525a093ea5bf40dc9bbf9dc39895d670: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:57.714322 18394 ts_tablet_manager.cc:1888] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 525a093ea5bf40dc9bbf9dc39895d670: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:29:57.715282 18394 log.cc:1196] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 525a093ea5bf40dc9bbf9dc39895d670: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/wals/9240163cfac446d2a14dbd9fcbbf9ff7
I20240502 21:29:57.716264  8949 catalog_manager.cc:4830] TS 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): tablet 9240163cfac446d2a14dbd9fcbbf9ff7 (table rebalance_test_table_0 [id=445d2e6ec7154b80abec5fff346b8f5e]) successfully deleted
I20240502 21:29:57.746729  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:29:57.748083  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:57.786273 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:57.788733 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:57.802496 11209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:57.813074 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:57.811623  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:29:57.850528 11228 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4d1a31513bb64578a85adf1f622e5bc0"
dest_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:55634
I20240502 21:29:57.850745 11228 raft_consensus.cc:595] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 LEADER]: Received request to transfer leadership
I20240502 21:29:57.931376  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:57.956662  9378 consensus_queue.cc:237] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:57.988935 11229 raft_consensus.cc:1261] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 21:29:57.990229 18331 consensus_queue.cc:1035] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:57.990814 11495 raft_consensus.cc:1261] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 21:29:57.991655 18331 consensus_queue.cc:1035] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:57.999789 18331 raft_consensus.cc:2932] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:58.004456 11229 raft_consensus.cc:2932] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:58.011025 11495 raft_consensus.cc:2932] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:58.033406  8953 catalog_manager.cc:5484] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: config changed from index 7 to 8, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New cstate: current_term: 2 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:58.046370  9221 tablet_service.cc:1514] Processing DeleteTablet for tablet f82c35077224414cb284472d49d7de4f with delete_type TABLET_DATA_TOMBSTONED (TS 525a093ea5bf40dc9bbf9dc39895d670 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:44408
I20240502 21:29:58.046860 18394 tablet_replica.cc:330] T f82c35077224414cb284472d49d7de4f P 525a093ea5bf40dc9bbf9dc39895d670: stopping tablet replica
I20240502 21:29:58.047089 18394 raft_consensus.cc:2229] T f82c35077224414cb284472d49d7de4f P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:58.047478 18394 raft_consensus.cc:2258] T f82c35077224414cb284472d49d7de4f P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:58.048437 18394 ts_tablet_manager.cc:1875] T f82c35077224414cb284472d49d7de4f P 525a093ea5bf40dc9bbf9dc39895d670: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:58.057273 18394 ts_tablet_manager.cc:1888] T f82c35077224414cb284472d49d7de4f P 525a093ea5bf40dc9bbf9dc39895d670: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 21:29:58.065608 18394 log.cc:1196] T f82c35077224414cb284472d49d7de4f P 525a093ea5bf40dc9bbf9dc39895d670: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/wals/f82c35077224414cb284472d49d7de4f
I20240502 21:29:58.069007  8949 catalog_manager.cc:4830] TS 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): tablet f82c35077224414cb284472d49d7de4f (table rebalance_test_table_5 [id=e8eef05f850243d0bdc7e1899f8c6139]) successfully deleted
W20240502 21:29:58.085492  9310 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.71:35857, user_credentials={real_user=slave}} blocked reactor thread for 67231.7us
I20240502 21:29:58.103489 18450 raft_consensus.cc:979] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe: : Instructing follower 5543ddab5b484642b139b7c848932a58 to start an election
I20240502 21:29:58.103737 18210 raft_consensus.cc:1067] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 LEADER]: Signalling peer 5543ddab5b484642b139b7c848932a58 to start an election
I20240502 21:29:58.104671  9105 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4d1a31513bb64578a85adf1f622e5bc0"
dest_uuid: "5543ddab5b484642b139b7c848932a58"
 from {username='slave'} at 127.27.196.70:53767
I20240502 21:29:58.104954  9105 raft_consensus.cc:482] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:29:58.105073  9105 raft_consensus.cc:3037] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:58.108135  9105 raft_consensus.cc:504] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:58.109724 11229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1a31513bb64578a85adf1f622e5bc0" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe"
I20240502 21:29:58.110666 11229 raft_consensus.cc:3032] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:29:58.110795 11229 raft_consensus.cc:729] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6ef0beaf2f6941afb6ea9f9f274517fe, State: Running, Role: LEADER
I20240502 21:29:58.111083 11229 consensus_queue.cc:260] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:58.111578 11229 raft_consensus.cc:3037] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:58.117228 11229 raft_consensus.cc:2454] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 3.
I20240502 21:29:58.117794  9039 leader_election.cc:302] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5543ddab5b484642b139b7c848932a58, 6ef0beaf2f6941afb6ea9f9f274517fe; no voters: 
I20240502 21:29:58.118103 18364 raft_consensus.cc:2781] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:29:58.118203 18364 raft_consensus.cc:686] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 3 LEADER]: Becoming Leader. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Running, Role: LEADER
I20240502 21:29:58.118469 18364 consensus_queue.cc:237] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:58.123910  9105 leader_election.cc:288] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [CANDIDATE]: Term 3 election: Requested vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467)
I20240502 21:29:58.124343 11495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1a31513bb64578a85adf1f622e5bc0" candidate_uuid: "5543ddab5b484642b139b7c848932a58" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "e0de0545fa0547d3929111e990e6d1b2"
I20240502 21:29:58.124572 11495 raft_consensus.cc:3037] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:29:58.131461  8953 catalog_manager.cc:5484] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 reported cstate change: term changed from 2 to 3, leader changed from 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) to 5543ddab5b484642b139b7c848932a58 (127.27.196.65). New cstate: current_term: 3 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:58.150699 11495 raft_consensus.cc:2454] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5543ddab5b484642b139b7c848932a58 in term 3.
I20240502 21:29:58.620293 11229 raft_consensus.cc:1261] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 3 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 21:29:58.623023 18364 consensus_queue.cc:1035] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:58.664008  9508 consensus_queue.cc:237] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:29:58.675029 11365 raft_consensus.cc:1261] T 07d693b680a54e12871121737e3da744 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 3 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 21:29:58.678140 11093 raft_consensus.cc:1261] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4 [term 3 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 21:29:58.680394 11495 raft_consensus.cc:1261] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 21:29:58.681629 18360 consensus_queue.cc:1035] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:58.687630 18005 consensus_queue.cc:1035] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:58.688582 18005 consensus_queue.cc:1035] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:58.781538  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:58.794170 18005 raft_consensus.cc:2932] T 07d693b680a54e12871121737e3da744 P 19f162e164854f2da623a4bb8f9e6bf6 [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:58.803189 11092 raft_consensus.cc:2932] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:58.812991  8953 catalog_manager.cc:5484] T 07d693b680a54e12871121737e3da744 P d10e9e156ba74a97a221c85b68ab8bc4 reported cstate change: config changed from index 9 to 11, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New cstate: current_term: 3 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:58.820252 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:58.827793 11364 raft_consensus.cc:2932] T 07d693b680a54e12871121737e3da744 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:29:58.867197  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:29:58.887367  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:29:58.968483 11209 tablet_service.cc:1514] Processing DeleteTablet for tablet 07d693b680a54e12871121737e3da744 with delete_type TABLET_DATA_TOMBSTONED (TS 6ef0beaf2f6941afb6ea9f9f274517fe not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:36872
I20240502 21:29:58.974211  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:59.001299 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:59.007905 18622 tablet_replica.cc:330] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe: stopping tablet replica
I20240502 21:29:59.008250 18622 raft_consensus.cc:2229] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:59.008562 18622 raft_consensus.cc:2258] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:59.009722 18622 ts_tablet_manager.cc:1875] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:59.037050 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:59.039449 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:59.058003 18622 ts_tablet_manager.cc:1888] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.10
I20240502 21:29:59.066380 18622 log.cc:1196] T 07d693b680a54e12871121737e3da744 P 6ef0beaf2f6941afb6ea9f9f274517fe: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-5/wal/wals/07d693b680a54e12871121737e3da744
I20240502 21:29:59.069355  8946 catalog_manager.cc:4830] TS 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): tablet 07d693b680a54e12871121737e3da744 (table rebalance_test_table_1 [id=f52c6c3804274777abfa00bb33ae9b16]) successfully deleted
I20240502 21:29:59.130162  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:29:59.142355  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:59.150532 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:59.177541 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:59.188740 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:59.205296  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:59.251991  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:29:59.271880 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:59.826018  9105 consensus_queue.cc:237] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:59.829497 11496 raft_consensus.cc:1261] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240502 21:29:59.842594 18364 consensus_queue.cc:1035] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:59.849916 18780 raft_consensus.cc:2932] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:59.850824  9508 consensus_queue.cc:237] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:29:59.860878 11496 raft_consensus.cc:2932] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:59.868587 11093 raft_consensus.cc:1261] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:29:59.869608 18595 consensus_queue.cc:1035] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:59.880550  8956 catalog_manager.cc:5484] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 reported cstate change: config changed from index 10 to 12, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New cstate: current_term: 3 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:29:59.911816 18595 raft_consensus.cc:2932] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:59.917191 11093 raft_consensus.cc:2932] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:59.920681  8963 catalog_manager.cc:5484] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: config changed from index 6 to 7, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New cstate: current_term: 2 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:29:59.954599 11209 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d1a31513bb64578a85adf1f622e5bc0 with delete_type TABLET_DATA_TOMBSTONED (TS 6ef0beaf2f6941afb6ea9f9f274517fe not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:36872
I20240502 21:29:59.955333  9105 consensus_queue.cc:237] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:59.960197 11496 raft_consensus.cc:1261] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 21:29:59.961575 18782 consensus_queue.cc:1035] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:59.963650 11228 raft_consensus.cc:1261] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 3 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 21:29:59.965484 18780 raft_consensus.cc:2932] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:59.968258 11496 raft_consensus.cc:2932] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:59.969756  9085 tablet_service.cc:1514] Processing DeleteTablet for tablet be34ec89876d42b69efe1878c0d9bc19 with delete_type TABLET_DATA_TOMBSTONED (TS 5543ddab5b484642b139b7c848932a58 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47814
I20240502 21:29:59.973718  9508 consensus_queue.cc:237] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:59.977306 11092 raft_consensus.cc:1261] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240502 21:29:59.978482  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: be34ec89876d42b69efe1878c0d9bc19. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:59.978818 18595 consensus_queue.cc:1035] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:59.981295 18782 consensus_queue.cc:1035] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:59.989032  8945 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 4d1a31513bb64578a85adf1f622e5bc0 with cas_config_opid_index 12: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 21:29:59.993198  8962 catalog_manager.cc:5484] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 reported cstate change: config changed from index 12 to 13, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New cstate: current_term: 3 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:29:59.997572 11228 raft_consensus.cc:858] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 3 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }.
  Pending config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }: 
  New config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
W20240502 21:29:59.997750 11228 raft_consensus.cc:1565] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 3 NON_PARTICIPANT]: Could not prepare op 'term: 3 index: 13' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }.
  Pending config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }: 
  New config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:30:00.001171 18595 raft_consensus.cc:2932] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:30:00.006562 11092 raft_consensus.cc:2932] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:30:00.011595  8945 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet be34ec89876d42b69efe1878c0d9bc19 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 21:30:00.019977 18808 tablet_replica.cc:330] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58: stopping tablet replica
I20240502 21:30:00.020241 18808 raft_consensus.cc:2229] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:30:00.020504 18808 raft_consensus.cc:2258] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:30:00.021353 18808 ts_tablet_manager.cc:1875] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:30:00.024212 11228 raft_consensus.cc:2932] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 3 NON_PARTICIPANT]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:30:00.037153  8963 catalog_manager.cc:5484] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4 reported cstate change: config changed from index 7 to 8, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New cstate: current_term: 2 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:30:00.050905 18808 ts_tablet_manager.cc:1888] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 21:30:00.056332 18808 log.cc:1196] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/wals/be34ec89876d42b69efe1878c0d9bc19
I20240502 21:30:00.058108  8945 catalog_manager.cc:4830] TS 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): tablet be34ec89876d42b69efe1878c0d9bc19 (table rebalance_test_table_6 [id=265d5656d7c048aa9795a906661b8ea8]) successfully deleted
W20240502 21:30:00.097018  8946 catalog_manager.cc:4804] TS 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): delete failed for tablet 4d1a31513bb64578a85adf1f622e5bc0 because of a CAS failure. No further retry: Illegal state: Request specified cas_config_opid_index_less_or_equal of 10 but the committed config has opid_index of 12
I20240502 21:30:00.097935 11228 raft_consensus.cc:1467] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 3 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20240502 21:30:00.099581 11228 raft_consensus.cc:2932] T 4d1a31513bb64578a85adf1f622e5bc0 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:30:00.400650  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:00.513456  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:30:00.531044  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:30:00.532631 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:30:00.575783 18930 ts_tablet_manager.cc:902] T be34ec89876d42b69efe1878c0d9bc19 P 6ef0beaf2f6941afb6ea9f9f274517fe: Initiating tablet copy from peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
I20240502 21:30:00.589524 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:00.601912 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:30:00.635566 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:00.692121  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:30:00.723330 18930 tablet_copy_client.cc:308] T be34ec89876d42b69efe1878c0d9bc19 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.68:40931
I20240502 21:30:00.724269  9533 tablet_copy_service.cc:140] P 19f162e164854f2da623a4bb8f9e6bf6: Received BeginTabletCopySession request for tablet be34ec89876d42b69efe1878c0d9bc19 from peer 6ef0beaf2f6941afb6ea9f9f274517fe ({username='slave'} at 127.27.196.70:53371)
I20240502 21:30:00.724435  9533 tablet_copy_service.cc:161] P 19f162e164854f2da623a4bb8f9e6bf6: Beginning new tablet copy session on tablet be34ec89876d42b69efe1878c0d9bc19 from peer 6ef0beaf2f6941afb6ea9f9f274517fe at {username='slave'} at 127.27.196.70:53371: session id = 6ef0beaf2f6941afb6ea9f9f274517fe-be34ec89876d42b69efe1878c0d9bc19
I20240502 21:30:00.736275  9533 tablet_copy_source_session.cc:205] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:30:00.738610 18930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be34ec89876d42b69efe1878c0d9bc19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:30:00.769224 18930 tablet_copy_client.cc:783] T be34ec89876d42b69efe1878c0d9bc19 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 0 data blocks...
I20240502 21:30:00.771723 18930 tablet_copy_client.cc:647] T be34ec89876d42b69efe1878c0d9bc19 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Starting download of 1 WAL segments...
I20240502 21:30:00.778939 18930 tablet_copy_client.cc:523] T be34ec89876d42b69efe1878c0d9bc19 P 6ef0beaf2f6941afb6ea9f9f274517fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:30:00.810879 18930 tablet_bootstrap.cc:492] T be34ec89876d42b69efe1878c0d9bc19 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap starting.
W20240502 21:30:01.009363  9313 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.72:40467, user_credentials={real_user=slave}} blocked reactor thread for 103244us
I20240502 21:30:01.032984 18930 tablet_bootstrap.cc:492] T be34ec89876d42b69efe1878c0d9bc19 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:30:01.033447 18930 tablet_bootstrap.cc:492] T be34ec89876d42b69efe1878c0d9bc19 P 6ef0beaf2f6941afb6ea9f9f274517fe: Bootstrap complete.
I20240502 21:30:01.033694 18930 ts_tablet_manager.cc:1367] T be34ec89876d42b69efe1878c0d9bc19 P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent bootstrapping tablet: real 0.223s	user 0.063s	sys 0.010s
I20240502 21:30:01.034448 18930 raft_consensus.cc:348] T be34ec89876d42b69efe1878c0d9bc19 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:30:01.034669 18930 raft_consensus.cc:729] T be34ec89876d42b69efe1878c0d9bc19 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef0beaf2f6941afb6ea9f9f274517fe, State: Initialized, Role: FOLLOWER
I20240502 21:30:01.034888 18930 consensus_queue.cc:260] T be34ec89876d42b69efe1878c0d9bc19 P 6ef0beaf2f6941afb6ea9f9f274517fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:30:01.084468 18930 ts_tablet_manager.cc:1398] T be34ec89876d42b69efe1878c0d9bc19 P 6ef0beaf2f6941afb6ea9f9f274517fe: Time spent starting tablet: real 0.051s	user 0.021s	sys 0.029s
I20240502 21:30:01.085426  9533 tablet_copy_service.cc:342] P 19f162e164854f2da623a4bb8f9e6bf6: Request end of tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-be34ec89876d42b69efe1878c0d9bc19 received from {username='slave'} at 127.27.196.70:53371
I20240502 21:30:01.085559  9533 tablet_copy_service.cc:433] P 19f162e164854f2da623a4bb8f9e6bf6: ending tablet copy session 6ef0beaf2f6941afb6ea9f9f274517fe-be34ec89876d42b69efe1878c0d9bc19 on tablet be34ec89876d42b69efe1878c0d9bc19 with peer 6ef0beaf2f6941afb6ea9f9f274517fe
I20240502 21:30:01.378984  9239 consensus_queue.cc:237] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:30:01.383879 11364 raft_consensus.cc:1261] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:30:01.384135  9378 raft_consensus.cc:1261] T 8129d13163f64368b731ba4eaeaaa81c P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:30:01.385015 18943 consensus_queue.cc:1035] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:30:01.385766 18943 consensus_queue.cc:1035] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:30:01.455507 18943 raft_consensus.cc:2932] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
W20240502 21:30:01.465687  9174 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.67:43203, user_credentials={real_user=slave}} blocked reactor thread for 80957us
I20240502 21:30:01.470994 11229 raft_consensus.cc:1203] T be34ec89876d42b69efe1878c0d9bc19 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8]   Dedup: 2.8->[]
I20240502 21:30:01.522380  9378 raft_consensus.cc:2932] T 8129d13163f64368b731ba4eaeaaa81c P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:30:01.540042  8955 catalog_manager.cc:5484] T 8129d13163f64368b731ba4eaeaaa81c P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: config changed from index 4 to 5, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:30:01.552466 11364 raft_consensus.cc:2932] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:30:01.689491  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:30:01.721237 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:01.723477 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:01.729097 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:30:01.745553  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:30:01.754503  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:30:01.766229 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:30:01.802502  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:30:01.809109  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:01.825346  9490 tablet_service.cc:1514] Processing DeleteTablet for tablet 8129d13163f64368b731ba4eaeaaa81c with delete_type TABLET_DATA_TOMBSTONED (TS 19f162e164854f2da623a4bb8f9e6bf6 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54670
I20240502 21:30:01.834415 19130 tablet_replica.cc:330] T 8129d13163f64368b731ba4eaeaaa81c P 19f162e164854f2da623a4bb8f9e6bf6: stopping tablet replica
I20240502 21:30:01.834791 19130 raft_consensus.cc:2229] T 8129d13163f64368b731ba4eaeaaa81c P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:30:01.835338 19130 raft_consensus.cc:2258] T 8129d13163f64368b731ba4eaeaaa81c P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:30:01.840226 19130 ts_tablet_manager.cc:1875] T 8129d13163f64368b731ba4eaeaaa81c P 19f162e164854f2da623a4bb8f9e6bf6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:30:01.843951 11209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:30:01.856822 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:30:01.890425 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:01.891271 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:01.906008  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:01.918980  9356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:30:01.928556  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:30:01.952431 19130 ts_tablet_manager.cc:1888] T 8129d13163f64368b731ba4eaeaaa81c P 19f162e164854f2da623a4bb8f9e6bf6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:30:01.952656 19130 log.cc:1196] T 8129d13163f64368b731ba4eaeaaa81c P 19f162e164854f2da623a4bb8f9e6bf6: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/wals/8129d13163f64368b731ba4eaeaaa81c
I20240502 21:30:01.959190  8945 catalog_manager.cc:4830] TS 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): tablet 8129d13163f64368b731ba4eaeaaa81c (table rebalance_test_table_9 [id=16e8100140034581b2a773dcf740998d]) successfully deleted
W20240502 21:30:02.183403  9445 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 64833.7us
W20240502 21:30:02.215730 11028 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.66:39071, user_credentials={real_user=slave}} blocked reactor thread for 50309.8us
I20240502 21:30:02.252425  9378 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f82c35077224414cb284472d49d7de4f"
dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:59194
I20240502 21:30:02.252765  9378 raft_consensus.cc:595] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 LEADER]: Received request to transfer leadership
I20240502 21:30:02.298233 18963 raft_consensus.cc:979] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0: : Instructing follower 6ef0beaf2f6941afb6ea9f9f274517fe to start an election
I20240502 21:30:02.298393 18963 raft_consensus.cc:1067] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 LEADER]: Signalling peer 6ef0beaf2f6941afb6ea9f9f274517fe to start an election
I20240502 21:30:02.299440 11229 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f82c35077224414cb284472d49d7de4f"
dest_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe"
 from {username='slave'} at 127.27.196.67:51767
I20240502 21:30:02.299712 11229 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:30:02.299856 11229 raft_consensus.cc:3037] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:30:02.303694 11229 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:30:02.304968 11229 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe [CANDIDATE]: Term 3 election: Requested vote from peers f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467)
I20240502 21:30:02.305536  9378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f82c35077224414cb284472d49d7de4f" candidate_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: true dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
I20240502 21:30:02.305845  9378 raft_consensus.cc:3032] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:30:02.305935  9378 raft_consensus.cc:729] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Running, Role: LEADER
I20240502 21:30:02.306298  9378 consensus_queue.cc:260] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:30:02.306838  9378 raft_consensus.cc:3037] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:30:02.311735 11496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f82c35077224414cb284472d49d7de4f" candidate_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: true dest_uuid: "e0de0545fa0547d3929111e990e6d1b2"
I20240502 21:30:02.312062 11496 raft_consensus.cc:3037] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:30:02.325253  9378 raft_consensus.cc:2454] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef0beaf2f6941afb6ea9f9f274517fe in term 3.
I20240502 21:30:02.326325 11162 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ef0beaf2f6941afb6ea9f9f274517fe, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:30:02.328078 19181 raft_consensus.cc:2781] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:30:02.328413 11496 raft_consensus.cc:2454] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef0beaf2f6941afb6ea9f9f274517fe in term 3.
I20240502 21:30:02.363240 19181 raft_consensus.cc:686] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe [term 3 LEADER]: Becoming Leader. State: Replica: 6ef0beaf2f6941afb6ea9f9f274517fe, State: Running, Role: LEADER
I20240502 21:30:02.364024 19181 consensus_queue.cc:237] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:30:02.393519  8955 catalog_manager.cc:5484] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe reported cstate change: term changed from 2 to 3, leader changed from f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) to 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70). New cstate: current_term: 3 leader_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:30:02.477073  9378 consensus_queue.cc:237] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:30:02.481412 11228 raft_consensus.cc:1261] T 4674e376bf7440cda619784491685b44 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:30:02.482949 18963 consensus_queue.cc:1035] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:30:02.497220 18963 raft_consensus.cc:2932] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:30:02.499356 11228 raft_consensus.cc:2932] T 4674e376bf7440cda619784491685b44 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:30:02.508311  8956 catalog_manager.cc:5484] T 4674e376bf7440cda619784491685b44 P 6ef0beaf2f6941afb6ea9f9f274517fe reported cstate change: config changed from index 5 to 6, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:30:02.524252  9378 consensus_queue.cc:237] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:30:02.527441  9085 tablet_service.cc:1514] Processing DeleteTablet for tablet 4674e376bf7440cda619784491685b44 with delete_type TABLET_DATA_TOMBSTONED (TS 5543ddab5b484642b139b7c848932a58 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47814
I20240502 21:30:02.531291 11228 raft_consensus.cc:1261] T 4674e376bf7440cda619784491685b44 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 21:30:02.531525  9313 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467): Couldn't send request to peer e0de0545fa0547d3929111e990e6d1b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4674e376bf7440cda619784491685b44. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:30:02.533469 19086 consensus_queue.cc:1035] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:30:02.541795 19234 tablet_replica.cc:330] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58: stopping tablet replica
I20240502 21:30:02.542903 19234 raft_consensus.cc:2229] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:30:02.543542 19234 raft_consensus.cc:2258] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:30:02.553267 19234 ts_tablet_manager.cc:1875] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:30:02.559962 18963 raft_consensus.cc:2932] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:30:02.560675 11228 raft_consensus.cc:2932] T 4674e376bf7440cda619784491685b44 P 6ef0beaf2f6941afb6ea9f9f274517fe [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:30:02.568189  8948 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 4674e376bf7440cda619784491685b44 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 21:30:02.570397  8963 catalog_manager.cc:5484] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: config changed from index 6 to 7, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:30:02.621685 19234 ts_tablet_manager.cc:1888] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:30:02.632455 19234 log.cc:1196] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/wals/4674e376bf7440cda619784491685b44
I20240502 21:30:02.638922  8945 catalog_manager.cc:4830] TS 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): tablet 4674e376bf7440cda619784491685b44 (table rebalance_test_table_3 [id=a237d7bda970485480e98020c6ac738f]) successfully deleted
I20240502 21:30:02.758536 11496 raft_consensus.cc:1261] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Refusing update from remote peer 6ef0beaf2f6941afb6ea9f9f274517fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240502 21:30:02.764297 19181 consensus_queue.cc:1035] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:30:02.775732  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:02.850541 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:30:02.863634  9378 raft_consensus.cc:1261] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 3 FOLLOWER]: Refusing update from remote peer 6ef0beaf2f6941afb6ea9f9f274517fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240502 21:30:02.864717 19181 consensus_queue.cc:1035] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:30:02.899602  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:30:02.958688  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:30:02.967041 11209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:02.982712 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:03.000196 19312 ts_tablet_manager.cc:902] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2: Initiating tablet copy from peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:30:03.013900 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:03.020910  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:03.063480 19312 tablet_copy_client.cc:308] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:43203
I20240502 21:30:03.064476  9399 tablet_copy_service.cc:140] P f2fdd596a8b44bbc848fe40f9f07fbe0: Received BeginTabletCopySession request for tablet 4674e376bf7440cda619784491685b44 from peer e0de0545fa0547d3929111e990e6d1b2 ({username='slave'} at 127.27.196.72:46235)
I20240502 21:30:03.064740  9399 tablet_copy_service.cc:161] P f2fdd596a8b44bbc848fe40f9f07fbe0: Beginning new tablet copy session on tablet 4674e376bf7440cda619784491685b44 from peer e0de0545fa0547d3929111e990e6d1b2 at {username='slave'} at 127.27.196.72:46235: session id = e0de0545fa0547d3929111e990e6d1b2-4674e376bf7440cda619784491685b44
I20240502 21:30:03.069461  9399 tablet_copy_source_session.cc:205] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:30:03.071375 19312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4674e376bf7440cda619784491685b44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:30:03.110257 19312 tablet_copy_client.cc:783] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Starting download of 0 data blocks...
I20240502 21:30:03.110711 19312 tablet_copy_client.cc:647] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:30:03.155212 19312 tablet_copy_client.cc:523] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:30:03.195863 19312 tablet_bootstrap.cc:492] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2: Bootstrap starting.
I20240502 21:30:03.399063 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:30:03.406117  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:03.438354  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:30:03.450933 19312 tablet_bootstrap.cc:492] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:30:03.452210 19312 tablet_bootstrap.cc:492] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2: Bootstrap complete.
I20240502 21:30:03.455602 19312 ts_tablet_manager.cc:1367] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2: Time spent bootstrapping tablet: real 0.260s	user 0.077s	sys 0.000s
I20240502 21:30:03.457571 19312 raft_consensus.cc:348] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:30:03.458448 19312 raft_consensus.cc:729] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0de0545fa0547d3929111e990e6d1b2, State: Initialized, Role: FOLLOWER
I20240502 21:30:03.458912 19312 consensus_queue.cc:260] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:30:03.460162 19312 ts_tablet_manager.cc:1398] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240502 21:30:03.462476  9399 tablet_copy_service.cc:342] P f2fdd596a8b44bbc848fe40f9f07fbe0: Request end of tablet copy session e0de0545fa0547d3929111e990e6d1b2-4674e376bf7440cda619784491685b44 received from {username='slave'} at 127.27.196.72:46235
I20240502 21:30:03.462646  9399 tablet_copy_service.cc:433] P f2fdd596a8b44bbc848fe40f9f07fbe0: ending tablet copy session e0de0545fa0547d3929111e990e6d1b2-4674e376bf7440cda619784491685b44 on tablet 4674e376bf7440cda619784491685b44 with peer e0de0545fa0547d3929111e990e6d1b2
I20240502 21:30:03.489636  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:30:03.540577 11494 raft_consensus.cc:1203] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 21:30:03.600191 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:03.662741  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:03.673880 11209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:03.703732 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:03.754891  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:03.801069 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:30:03.801270  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:30:03.905153  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:30:03.946681 11209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:03.949086  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:03.949431 11494 consensus_queue.cc:237] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:30:03.956842 11093 raft_consensus.cc:1261] T a7c6b881c9ef4cdca787f065778e526f P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Refusing update from remote peer e0de0545fa0547d3929111e990e6d1b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:30:03.957856 18508 consensus_queue.cc:1035] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:30:03.967976 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:03.970265 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:03.983531 19454 raft_consensus.cc:2932] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:30:04.020417 11093 raft_consensus.cc:2932] T a7c6b881c9ef4cdca787f065778e526f P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:30:04.031057  8963 catalog_manager.cc:5484] T a7c6b881c9ef4cdca787f065778e526f P d10e9e156ba74a97a221c85b68ab8bc4 reported cstate change: config changed from index 8 to 9, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New cstate: current_term: 2 leader_uuid: "e0de0545fa0547d3929111e990e6d1b2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:30:04.098614  9221 tablet_service.cc:1514] Processing DeleteTablet for tablet a7c6b881c9ef4cdca787f065778e526f with delete_type TABLET_DATA_TOMBSTONED (TS 525a093ea5bf40dc9bbf9dc39895d670 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:44408
I20240502 21:30:04.100157 11494 consensus_queue.cc:237] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:30:04.103511 11093 raft_consensus.cc:1261] T a7c6b881c9ef4cdca787f065778e526f P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Refusing update from remote peer e0de0545fa0547d3929111e990e6d1b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
W20240502 21:30:04.106266 11430 consensus_peers.cc:485] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7c6b881c9ef4cdca787f065778e526f. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:30:04.106941 19458 consensus_queue.cc:1035] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:30:04.110107 19458 raft_consensus.cc:2932] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:30:04.114565 11093 raft_consensus.cc:2932] T a7c6b881c9ef4cdca787f065778e526f P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:30:04.118227 19478 tablet_replica.cc:330] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670: stopping tablet replica
I20240502 21:30:04.118525 19478 raft_consensus.cc:2229] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:30:04.118875 19478 raft_consensus.cc:2258] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:30:04.119897 19478 ts_tablet_manager.cc:1875] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:30:04.121181  8949 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet a7c6b881c9ef4cdca787f065778e526f with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 21:30:04.121537  8963 catalog_manager.cc:5484] T a7c6b881c9ef4cdca787f065778e526f P d10e9e156ba74a97a221c85b68ab8bc4 reported cstate change: config changed from index 9 to 10, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New cstate: current_term: 2 leader_uuid: "e0de0545fa0547d3929111e990e6d1b2" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:30:04.128743 19478 ts_tablet_manager.cc:1888] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240502 21:30:04.128949 19478 log.cc:1196] T a7c6b881c9ef4cdca787f065778e526f P 525a093ea5bf40dc9bbf9dc39895d670: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/wals/a7c6b881c9ef4cdca787f065778e526f
I20240502 21:30:04.130475  8949 catalog_manager.cc:4830] TS 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): tablet a7c6b881c9ef4cdca787f065778e526f (table rebalance_test_table_6 [id=265d5656d7c048aa9795a906661b8ea8]) successfully deleted
I20240502 21:30:04.207660  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:04.236836  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:30:04.246008 11343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:30:04.283313 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:04.290208  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:30:04.314782 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:04.353031 11209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:04.396646  9377 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f0259f01acb249dd99c662151dc21916"
dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:59254
I20240502 21:30:04.397002  9377 raft_consensus.cc:595] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 LEADER]: Received request to transfer leadership
I20240502 21:30:04.421033  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:04.495522 11228 consensus_queue.cc:237] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:30:04.526592 11496 raft_consensus.cc:1261] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Refusing update from remote peer 6ef0beaf2f6941afb6ea9f9f274517fe: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 21:30:04.528344 19181 consensus_queue.cc:1035] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:30:04.533161 19533 ts_tablet_manager.cc:902] T a7c6b881c9ef4cdca787f065778e526f P 6ef0beaf2f6941afb6ea9f9f274517fe: Initiating tablet copy from peer e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467)
F20240502 21:30:04.538911 19533 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-07d693b680a54e12871121737e3da744.queue_time_us, tablet-download-pool-07d693b680a54e12871121737e3da744.queue_time_us) (tablet-download-pool-07d693b680a54e12871121737e3da744.run_cpu_time_us, tablet-download-pool-07d693b680a54e12871121737e3da744.run_cpu_time_us) (tablet-download-pool-07d693b680a54e12871121737e3da744.run_wall_time_us, tablet-download-pool-07d693b680a54e12871121737e3da744.run_wall_time_us) (tablet-download-pool-32c306d724954a16be21a34aa4cc0b5f.queue_time_us, tablet-download-pool-32c306d724954a16be21a34aa4cc0b5f.queue_time_us) (tablet-download-pool-32c306d724954a16be21a34aa4cc0b5f.run_cpu_time_us, tablet-download-pool-32c306d724954a16be21a34aa4cc0b5f.run_cpu_time_us) (tablet-download-pool-32c306d724954a16be21a34aa4cc0b5f.run_wall_time_us, tablet-download-pool-32c306d724954a16be21a34aa4cc0b5f.run_wall_time_us) (tablet-download-pool-3caa36212ba444aa88d153ff9f90122e.queue_time_us, tablet-download-pool-3caa36212ba444aa88d153ff9f90122e.queue_time_us) (tablet-download-pool-3caa36212ba444aa88d153ff9f90122e.run_cpu_time_us, tablet-download-pool-3caa36212ba444aa88d153ff9f90122e.run_cpu_time_us) (tablet-download-pool-3caa36212ba444aa88d153ff9f90122e.run_wall_time_us, tablet-download-pool-3caa36212ba444aa88d153ff9f90122e.run_wall_time_us) (tablet-download-pool-3d1a5050d3f944969a560994a23700ad.queue_time_us, tablet-download-pool-3d1a5050d3f944969a560994a23700ad.queue_time_us) (tablet-download-pool-3d1a5050d3f944969a560994a23700ad.run_cpu_time_us, tablet-download-pool-3d1a5050d3f944969a560994a23700ad.run_cpu_time_us) (tablet-download-pool-3d1a5050d3f944969a560994a23700ad.run_wall_time_us, tablet-download-pool-3d1a5050d3f944969a560994a23700ad.run_wall_time_us) (tablet-download-pool-4674e376bf7440cda619784491685b44.queue_time_us, tablet-download-pool-4674e376bf7440cda619784491685b44.queue_time_us) (tablet-download-pool-4674e376bf7440cda619784491685b44.run_cpu_time_us, tablet-download-pool-4674e376bf7440cda619784491685b44.run_cpu_time_us) (tablet-download-pool-4674e376bf7440cda619784491685b44.run_wall_time_us, tablet-download-pool-4674e376bf7440cda619784491685b44.run_wall_time_us) (tablet-download-pool-4d1a31513bb64578a85adf1f622e5bc0.queue_time_us, tablet-download-pool-4d1a31513bb64578a85adf1f622e5bc0.queue_time_us) (tablet-download-pool-4d1a31513bb64578a85adf1f622e5bc0.run_cpu_time_us, tablet-download-pool-4d1a31513bb64578a85adf1f622e5bc0.run_cpu_time_us) (tablet-download-pool-4d1a31513bb64578a85adf1f622e5bc0.run_wall_time_us, tablet-download-pool-4d1a31513bb64578a85adf1f622e5bc0.run_wall_time_us) (tablet-download-pool-58c7004ceb044f5fa9dec54910046b69.queue_time_us, tablet-download-pool-58c7004ceb044f5fa9dec54910046b69.queue_time_us) (tablet-download-pool-58c7004ceb044f5fa9dec54910046b69.run_cpu_time_us, tablet-download-pool-58c7004ceb044f5fa9dec54910046b69.run_cpu_time_us) (tablet-download-pool-58c7004ceb044f5fa9dec54910046b69.run_wall_time_us, tablet-download-pool-58c7004ceb044f5fa9dec54910046b69.run_wall_time_us) (tablet-download-pool-8a2e06253c51405fa9b6bc9e87844760.queue_time_us, tablet-download-pool-8a2e06253c51405fa9b6bc9e87844760.queue_time_us) (tablet-download-pool-8a2e06253c51405fa9b6bc9e87844760.run_cpu_time_us, tablet-download-pool-8a2e06253c51405fa9b6bc9e87844760.run_cpu_time_us) (tablet-download-pool-8a2e06253c51405fa9b6bc9e87844760.run_wall_time_us, tablet-download-pool-8a2e06253c51405fa9b6bc9e87844760.run_wall_time_us) (tablet-download-pool-9240163cfac446d2a14dbd9fcbbf9ff7.queue_time_us, tablet-download-pool-9240163cfac446d2a14dbd9fcbbf9ff7.queue_time_us) (tablet-download-pool-9240163cfac446d2a14dbd9fcbbf9ff7.run_cpu_time_us, tablet-download-pool-9240163cfac446d2a14dbd9fcbbf9ff7.run_cpu_time_us) (tablet-download-pool-9240163cfac446d2a14dbd9fcbbf9ff7.run_wall_time_us, tablet-download-pool-9240163cfac446d2a14dbd9fcbbf9ff7.run_wall_time_us) (tablet-download-pool-94d2cdf8ae6941668b5f680c558656e8.queue_time_us, tablet-download-pool-94d2cdf8ae6941668b5f680c558656e8.queue_time_us) (tablet-download-pool-94d2cdf8ae6941668b5f680c558656e8.run_cpu_time_us, tablet-download-pool-94d2cdf8ae6941668b5f680c558656e8.run_cpu_time_us) (tablet-download-pool-94d2cdf8ae6941668b5f680c558656e8.run_wall_time_us, tablet-download-pool-94d2cdf8ae6941668b5f680c558656e8.run_wall_time_us) (tablet-download-pool-a7c6b881c9ef4cdca787f065778e526f.queue_time_us, tablet-download-pool-a7c6b881c9ef4cdca787f065778e526f.queue_time_us) (tablet-download-pool-be34ec89876d42b69efe1878c0d9bc19.queue_time_us, tablet-download-pool-be34ec89876d42b69efe1878c0d9bc19.queue_time_us) (tablet-download-pool-be34ec89876d42b69efe1878c0d9bc19.run_cpu_time_us, tablet-download-pool-be34ec89876d42b69efe1878c0d9bc19.run_cpu_time_us) (tablet-download-pool-be34ec89876d42b69efe1878c0d9bc19.run_wall_time_us, tablet-download-pool-be34ec89876d42b69efe1878c0d9bc19.run_wall_time_us) (tablet-download-pool-c56ac926025e47218e9df4088184a278.queue_time_us, tablet-download-pool-c56ac926025e47218e9df4088184a278.queue_time_us) (tablet-download-pool-c56ac926025e47218e9df4088184a278.run_cpu_time_us, tablet-download-pool-c56ac926025e47218e9df4088184a278.run_cpu_time_us) (tablet-download-pool-c56ac926025e47218e9df4088184a278.run_wall_time_us, tablet-download-pool-c56ac926025e47218e9df4088184a278.run_wall_time_us) (tablet-download-pool-dc1970b9263d41abb575b9d8639c59ae.queue_time_us, tablet-download-pool-dc1970b9263d41abb575b9d8639c59ae.queue_time_us) (tablet-download-pool-dc1970b9263d41abb575b9d8639c59ae.run_cpu_time_us, tablet-download-pool-dc1970b9263d41abb575b9d8639c59ae.run_cpu_time_us) (tablet-download-pool-dc1970b9263d41abb575b9d8639c59ae.run_wall_time_us, tablet-download-pool-dc1970b9263d41abb575b9d8639c59ae.run_wall_time_us) (tablet-download-pool-f82c35077224414cb284472d49d7de4f.queue_time_us, tablet-download-pool-f82c35077224414cb284472d49d7de4f.queue_time_us) (tablet-download-pool-f82c35077224414cb284472d49d7de4f.run_cpu_time_us, tablet-download-pool-f82c35077224414cb284472d49d7de4f.run_cpu_time_us) (tablet-download-pool-f82c35077224414cb284472d49d7de4f.run_wall_time_us, tablet-download-pool-f82c35077224414cb284472d49d7de4f.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
    @     0x7fec9988edcd  google::LogMessage::Fail() at ??:0
    @     0x7fec99892b93  google::LogMessage::SendToLog() at ??:0
    @     0x7fec9988e7cc  google::LogMessage::Flush() at ??:0
    @     0x7fec9988ff59  google::LogMessageFatal::~LogMessageFatal() at ??:0
    @     0x7fec9c5cf742  kudu::TraceMetrics::InternName() at ??:0
    @     0x7fec9c59bf8b  kudu::ThreadPool::ThreadPool() at ??:0
I20240502 21:30:04.594249 19538 raft_consensus.cc:2932] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe [term 3 LEADER]: Committing config change with OpId 3.10: config changed from index 8 to 10, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:30:04.595207 11496 raft_consensus.cc:2932] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Committing config change with OpId 3.10: config changed from index 8 to 10, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
    @     0x7fec9c595cd6  kudu::ThreadPoolBuilder::Build() at ??:0
I20240502 21:30:04.602828  8953 catalog_manager.cc:5484] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 reported cstate change: config changed from index 8 to 10, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New cstate: current_term: 3 leader_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
    @     0x7fecab9e88de  kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
    @     0x7fecaba00daf  kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240502 21:30:04.614988  9356 tablet_service.cc:1514] Processing DeleteTablet for tablet f82c35077224414cb284472d49d7de4f with delete_type TABLET_DATA_TOMBSTONED (TS f2fdd596a8b44bbc848fe40f9f07fbe0 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55106
    @     0x7fecabb2b480  kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
    @     0x7fecabb63a8d  kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240502 21:30:04.630609 11228 consensus_queue.cc:237] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
    @     0x7fecabb46981  kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
W20240502 21:30:04.634522 11161 consensus_peers.cc:485] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe -> Peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Couldn't send request to peer d10e9e156ba74a97a221c85b68ab8bc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f82c35077224414cb284472d49d7de4f. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:30:04.635179 11496 raft_consensus.cc:1261] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Refusing update from remote peer 6ef0beaf2f6941afb6ea9f9f274517fe: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 21:30:04.636024 19538 consensus_queue.cc:1035] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
    @     0x7fecabb4670e  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
I20240502 21:30:04.641418 19538 raft_consensus.cc:2932] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:30:04.644892 19549 tablet_replica.cc:330] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0: stopping tablet replica
I20240502 21:30:04.645318 19549 raft_consensus.cc:2229] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:30:04.645501 19549 raft_consensus.cc:2258] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:30:04.652097 11496 raft_consensus.cc:2932] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:30:04.653532  8953 catalog_manager.cc:5484] T f82c35077224414cb284472d49d7de4f P 6ef0beaf2f6941afb6ea9f9f274517fe reported cstate change: config changed from index 10 to 11, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New cstate: current_term: 3 leader_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:30:04.657929  8946 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet f82c35077224414cb284472d49d7de4f with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
    @     0x7fec9c59fb75  kudu::ThreadPool::DispatchThread() at ??:0
    @     0x7fec9c5a319c  kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
    @     0x7fec9c5a3019  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7fec9c57e01e  kudu::Thread::SuperviseThread() at ??:0
    @     0x7fec9f9036db  start_thread at ??:0
I20240502 21:30:04.680020 19549 ts_tablet_manager.cc:1875] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
    @     0x7fec9874171f  clone at ??:0
I20240502 21:30:04.712024 19086 raft_consensus.cc:979] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0: : Instructing follower 4cbb5b06a6a44eb6b2145cd538b01af2 to start an election
I20240502 21:30:04.712209 19086 raft_consensus.cc:1067] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 LEADER]: Signalling peer 4cbb5b06a6a44eb6b2145cd538b01af2 to start an election
I20240502 21:30:04.713161 11364 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f0259f01acb249dd99c662151dc21916"
dest_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2"
 from {username='slave'} at 127.27.196.67:43725
I20240502 21:30:04.713922 11364 raft_consensus.cc:482] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:30:04.714172 11364 raft_consensus.cc:3037] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:30:04.724164 11364 raft_consensus.cc:504] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:30:04.726809  9508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0259f01acb249dd99c662151dc21916" candidate_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6"
I20240502 21:30:04.727066  9508 raft_consensus.cc:3037] T f0259f01acb249dd99c662151dc21916 P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:30:04.730502  9508 raft_consensus.cc:2454] T f0259f01acb249dd99c662151dc21916 P 19f162e164854f2da623a4bb8f9e6bf6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cbb5b06a6a44eb6b2145cd538b01af2 in term 3.
I20240502 21:30:04.731846 11295 leader_election.cc:302] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 4cbb5b06a6a44eb6b2145cd538b01af2; no voters: 
I20240502 21:30:04.746817  9377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0259f01acb249dd99c662151dc21916" candidate_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
I20240502 21:30:04.747171  9377 raft_consensus.cc:3032] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:30:04.747269  9377 raft_consensus.cc:729] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Running, Role: LEADER
I20240502 21:30:04.747611  9377 consensus_queue.cc:260] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:30:04.748041  9377 raft_consensus.cc:3037] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:30:04.752951  9377 raft_consensus.cc:2454] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cbb5b06a6a44eb6b2145cd538b01af2 in term 3.
I20240502 21:30:04.757133 11364 leader_election.cc:288] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 3 election: Requested vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:30:04.766960 19573 raft_consensus.cc:2781] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:30:04.793671 19573 raft_consensus.cc:686] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 3 LEADER]: Becoming Leader. State: Replica: 4cbb5b06a6a44eb6b2145cd538b01af2, State: Running, Role: LEADER
I20240502 21:30:04.794420 19573 consensus_queue.cc:237] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:30:04.804575 19549 ts_tablet_manager.cc:1888] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.9
I20240502 21:30:04.804754 19549 log.cc:1196] T f82c35077224414cb284472d49d7de4f P f2fdd596a8b44bbc848fe40f9f07fbe0: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/wals/f82c35077224414cb284472d49d7de4f
I20240502 21:30:04.805672  8948 catalog_manager.cc:4830] TS f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): tablet f82c35077224414cb284472d49d7de4f (table rebalance_test_table_5 [id=e8eef05f850243d0bdc7e1899f8c6139]) successfully deleted
I20240502 21:30:04.838084  8963 catalog_manager.cc:5484] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 reported cstate change: term changed from 2 to 3, leader changed from f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) to 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71). New cstate: current_term: 3 leader_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:30:04.914443  9105 consensus_queue.cc:237] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:30:04.938694 11496 raft_consensus.cc:1261] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240502 21:30:04.940300 18780 consensus_queue.cc:1035] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:30:04.949561 18782 raft_consensus.cc:2932] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 13 to 14, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:30:04.950843 11496 raft_consensus.cc:2932] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 13 to 14, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:30:04.962369  8963 catalog_manager.cc:5484] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index 13 to 14, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New cstate: current_term: 3 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:30:04.980973  9105 consensus_queue.cc:237] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:30:05.003582 11496 raft_consensus.cc:1261] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240502 21:30:05.004403 18780 consensus_queue.cc:1035] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20240502 21:30:05.007840  9039 consensus_peers.cc:485] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 -> Peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Couldn't send request to peer d10e9e156ba74a97a221c85b68ab8bc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4d1a31513bb64578a85adf1f622e5bc0. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:30:05.008963 18780 raft_consensus.cc:2932] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58 [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 14 to 15, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:30:05.012028 11496 raft_consensus.cc:2932] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 14 to 15, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:30:05.066906  8962 catalog_manager.cc:5484] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 reported cstate change: config changed from index 14 to 15, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New cstate: current_term: 3 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:30:05.080266  8945 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 4d1a31513bb64578a85adf1f622e5bc0 with cas_config_opid_index 14: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 21:30:05.167611  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:30:05.187470  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:05.232364  9508 raft_consensus.cc:1261] T f0259f01acb249dd99c662151dc21916 P 19f162e164854f2da623a4bb8f9e6bf6 [term 3 FOLLOWER]: Refusing update from remote peer 4cbb5b06a6a44eb6b2145cd538b01af2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240502 21:30:05.233443 19573 consensus_queue.cc:1035] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:30:05.319613 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:05.320504  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:30:05.326642  9356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:30:05.354385  9377 raft_consensus.cc:1261] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 3 FOLLOWER]: Refusing update from remote peer 4cbb5b06a6a44eb6b2145cd538b01af2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240502 21:30:05.355279 19579 consensus_queue.cc:1035] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 21:30:05.357661 11430 connection.cc:535] client connection to 127.27.196.70:44211 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:30:05.358003 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:30:05.358454  8946 connection.cc:535] client connection to 127.27.196.70:44211 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:30:05.358685  8946 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:30:05.359369  9311 connection.cc:535] client connection to 127.27.196.70:44211 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:30:05.359692  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:30:05.377126  9039 connection.cc:535] client connection to 127.27.196.70:44211 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:30:05.377437  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:30:05.378813  9173 connection.cc:535] client connection to 127.27.196.70:44211 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:30:05.379665  9445 connection.cc:535] client connection to 127.27.196.70:44211 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:30:05.380059  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:30:05.381888  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:30:05.390230  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 21:30:05.415905  8946 catalog_manager.cc:4573] TS 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4d1a31513bb64578a85adf1f622e5bc0: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:05.423568  9173 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 44532.7us
W20240502 21:30:05.424646  9039 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 47297.7us
W20240502 21:30:05.425220  9445 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 45286.6us
I20240502 21:30:05.449786 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:05.452872 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:05.460925  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:05.497768 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:05.497957 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:30:05.500471  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:30:05.500681  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:30:05.500806  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:30:05.505357  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:30:05.506229 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:05.507095  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:30:05.507297  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:30:05.511794  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:30:05.512077  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:30:05.512164  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:30:05.521507  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:30:05.566562 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:05.577625  9356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:30:05.581727 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:05.640920 19725 ts_tablet_manager.cc:902] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4: Initiating tablet copy from peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
W20240502 21:30:05.674911 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:05.747365 19725 tablet_copy_client.cc:308] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:42177
I20240502 21:30:05.748287  9128 tablet_copy_service.cc:140] P 5543ddab5b484642b139b7c848932a58: Received BeginTabletCopySession request for tablet 4d1a31513bb64578a85adf1f622e5bc0 from peer d10e9e156ba74a97a221c85b68ab8bc4 ({username='slave'} at 127.27.196.69:40153)
I20240502 21:30:05.748461  9128 tablet_copy_service.cc:161] P 5543ddab5b484642b139b7c848932a58: Beginning new tablet copy session on tablet 4d1a31513bb64578a85adf1f622e5bc0 from peer d10e9e156ba74a97a221c85b68ab8bc4 at {username='slave'} at 127.27.196.69:40153: session id = d10e9e156ba74a97a221c85b68ab8bc4-4d1a31513bb64578a85adf1f622e5bc0
I20240502 21:30:05.751854  9128 tablet_copy_source_session.cc:205] T 4d1a31513bb64578a85adf1f622e5bc0 P 5543ddab5b484642b139b7c848932a58: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:30:05.753751 19725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d1a31513bb64578a85adf1f622e5bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:30:05.788429  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:30:05.811012  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:05.834035  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:30:05.850921 19725 tablet_copy_client.cc:783] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Starting download of 0 data blocks...
I20240502 21:30:05.851296 19725 tablet_copy_client.cc:647] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:30:05.873864 19725 tablet_copy_client.cc:523] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:30:05.880450 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:05.890451 19725 tablet_bootstrap.cc:492] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap starting.
I20240502 21:30:05.898036 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:05.904320  9041 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.72:40467, user_credentials={real_user=slave}} blocked reactor thread for 46695.4us
I20240502 21:30:05.926851  9356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:30:05.959148 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:06.062027  9221 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 21:30:06.078435 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:06.115020 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:06.140249  9356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:30:06.152704 19458 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:30:06.152973 19458 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:30:06.154011 19458 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:30:06.158141 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
W20240502 21:30:06.160194 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:06.160351 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:30:06.161108 19458 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 21:30:06.162688 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:06.184032  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:06.197270 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:06.200369  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:30:06.406380 19725 tablet_bootstrap.cc:492] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:30:06.406911 19725 tablet_bootstrap.cc:492] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap complete.
I20240502 21:30:06.407375 19725 ts_tablet_manager.cc:1367] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4: Time spent bootstrapping tablet: real 0.517s	user 0.089s	sys 0.006s
I20240502 21:30:06.408560 19725 raft_consensus.cc:348] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:30:06.408905 19725 raft_consensus.cc:729] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: d10e9e156ba74a97a221c85b68ab8bc4, State: Initialized, Role: FOLLOWER
I20240502 21:30:06.409333 19725 consensus_queue.cc:260] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 3.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:30:06.415035 19725 ts_tablet_manager.cc:1398] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.000s
I20240502 21:30:06.420148  9128 tablet_copy_service.cc:342] P 5543ddab5b484642b139b7c848932a58: Request end of tablet copy session d10e9e156ba74a97a221c85b68ab8bc4-4d1a31513bb64578a85adf1f622e5bc0 received from {username='slave'} at 127.27.196.69:40153
I20240502 21:30:06.420347  9128 tablet_copy_service.cc:433] P 5543ddab5b484642b139b7c848932a58: ending tablet copy session d10e9e156ba74a97a221c85b68ab8bc4-4d1a31513bb64578a85adf1f622e5bc0 on tablet 4d1a31513bb64578a85adf1f622e5bc0 with peer d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:30:06.505939  9239 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "87f35aff2b104c01a9d9d48a7e9c9b23"
dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:54414
I20240502 21:30:06.506279  9239 raft_consensus.cc:595] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:30:06.526026  9105 consensus_queue.cc:237] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:30:06.539448 11365 raft_consensus.cc:1261] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:30:06.540486 19767 consensus_queue.cc:1035] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:30:06.563647 19767 raft_consensus.cc:2932] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:30:06.568063 11364 raft_consensus.cc:2932] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:30:06.574153  8963 catalog_manager.cc:5484] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index 5 to 6, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:30:06.615387  9221 tablet_service.cc:1514] Processing DeleteTablet for tablet cdf72d25f0f847e5998acbc53979e64d with delete_type TABLET_DATA_TOMBSTONED (TS 525a093ea5bf40dc9bbf9dc39895d670 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44408
I20240502 21:30:06.616302 19870 tablet_replica.cc:330] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670: stopping tablet replica
I20240502 21:30:06.617585 19870 raft_consensus.cc:2229] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:30:06.617069  9105 consensus_queue.cc:237] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:30:06.617949 19870 raft_consensus.cc:2258] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:30:06.619042 19870 ts_tablet_manager.cc:1875] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:30:06.626607 19870 ts_tablet_manager.cc:1888] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:30:06.626904 19870 log.cc:1196] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/wals/cdf72d25f0f847e5998acbc53979e64d
I20240502 21:30:06.628069  8949 catalog_manager.cc:4830] TS 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): tablet cdf72d25f0f847e5998acbc53979e64d (table rebalance_test_table_2 [id=c0a0ef39a2a246449eb582cc1d993484]) successfully deleted
W20240502 21:30:06.630841  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:30:06.632517 11365 raft_consensus.cc:1261] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 21:30:06.633075 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:06.633754 19767 consensus_queue.cc:1035] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:30:06.641297 19767 raft_consensus.cc:2932] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:30:06.672262 11364 raft_consensus.cc:2932] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
I20240502 21:30:06.682565  8945 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet cdf72d25f0f847e5998acbc53979e64d with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 21:30:06.687203  8963 catalog_manager.cc:5484] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index 6 to 7, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) added. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } }
W20240502 21:30:06.697798  9444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.71:35857, user_credentials={real_user=slave}} blocked reactor thread for 45420.7us
I20240502 21:30:06.721459 11092 raft_consensus.cc:1203] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.14->[3.15-3.15]   Dedup: 3.15->[]
W20240502 21:30:06.747189  8946 catalog_manager.cc:4573] TS 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4d1a31513bb64578a85adf1f622e5bc0: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:06.836978 19891 raft_consensus.cc:979] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670: : Instructing follower f2fdd596a8b44bbc848fe40f9f07fbe0 to start an election
I20240502 21:30:06.837152 19891 raft_consensus.cc:1067] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Signalling peer f2fdd596a8b44bbc848fe40f9f07fbe0 to start an election
I20240502 21:30:06.853912  9377 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "87f35aff2b104c01a9d9d48a7e9c9b23"
dest_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0"
 from {username='slave'} at 127.27.196.66:49721
I20240502 21:30:06.854283  9377 raft_consensus.cc:482] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:30:06.854413  9377 raft_consensus.cc:3037] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:30:06.858889  9377 raft_consensus.cc:504] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:30:06.863266  9239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f35aff2b104c01a9d9d48a7e9c9b23" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "525a093ea5bf40dc9bbf9dc39895d670"
I20240502 21:30:06.863611  9239 raft_consensus.cc:3032] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:30:06.863706  9239 raft_consensus.cc:729] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Running, Role: LEADER
I20240502 21:30:06.864096  9239 consensus_queue.cc:260] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:30:06.864586  9239 raft_consensus.cc:3037] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:30:06.869154  9239 raft_consensus.cc:2454] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 2.
I20240502 21:30:06.870148  9313 leader_election.cc:302] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525a093ea5bf40dc9bbf9dc39895d670, f2fdd596a8b44bbc848fe40f9f07fbe0; no voters: 
I20240502 21:30:06.870667 19686 raft_consensus.cc:2781] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:30:06.875756 11365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f35aff2b104c01a9d9d48a7e9c9b23" candidate_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2"
I20240502 21:30:06.876121 11365 raft_consensus.cc:3037] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:30:06.881108 11365 raft_consensus.cc:2454] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fdd596a8b44bbc848fe40f9f07fbe0 in term 2.
I20240502 21:30:06.882594  9377 leader_election.cc:288] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 [CANDIDATE]: Term 2 election: Requested vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857)
I20240502 21:30:06.897172 19686 raft_consensus.cc:686] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 LEADER]: Becoming Leader. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Running, Role: LEADER
I20240502 21:30:06.897877 19686 consensus_queue.cc:237] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:30:06.901170 11365 consensus_queue.cc:237] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:30:06.910348  8953 catalog_manager.cc:5484] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: term changed from 1 to 2, leader changed from 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) to f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67). New cstate: current_term: 2 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:30:06.975955  9507 raft_consensus.cc:1261] T f0259f01acb249dd99c662151dc21916 P 19f162e164854f2da623a4bb8f9e6bf6 [term 3 FOLLOWER]: Refusing update from remote peer 4cbb5b06a6a44eb6b2145cd538b01af2: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240502 21:30:06.977365 19573 consensus_queue.cc:1035] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:30:07.007136  9507 raft_consensus.cc:2932] T f0259f01acb249dd99c662151dc21916 P 19f162e164854f2da623a4bb8f9e6bf6 [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 6 to 8, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:30:07.014292  8963 catalog_manager.cc:5484] T f0259f01acb249dd99c662151dc21916 P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: config changed from index 6 to 8, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New cstate: current_term: 3 leader_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:30:07.021152  9356 tablet_service.cc:1514] Processing DeleteTablet for tablet f0259f01acb249dd99c662151dc21916 with delete_type TABLET_DATA_TOMBSTONED (TS f2fdd596a8b44bbc848fe40f9f07fbe0 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:55106
W20240502 21:30:07.040797 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:07.005511 19915 raft_consensus.cc:2932] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 3 LEADER]: Committing config change with OpId 3.8: config changed from index 6 to 8, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:30:07.059783 19921 tablet_replica.cc:330] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0: stopping tablet replica
I20240502 21:30:07.060149 19921 raft_consensus.cc:2229] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:30:07.060422 19921 raft_consensus.cc:2258] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:30:07.061354 19921 ts_tablet_manager.cc:1875] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 21:30:07.079320 11295 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.68:40931, user_credentials={real_user=slave}} blocked reactor thread for 66412.2us
I20240502 21:30:07.087792 19921 ts_tablet_manager.cc:1888] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.7
I20240502 21:30:07.087965 11365 consensus_queue.cc:237] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:30:07.091611  9507 raft_consensus.cc:1261] T f0259f01acb249dd99c662151dc21916 P 19f162e164854f2da623a4bb8f9e6bf6 [term 3 FOLLOWER]: Refusing update from remote peer 4cbb5b06a6a44eb6b2145cd538b01af2: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
W20240502 21:30:07.093094 11298 consensus_peers.cc:485] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 -> Peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Couldn't send request to peer 525a093ea5bf40dc9bbf9dc39895d670. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f0259f01acb249dd99c662151dc21916. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:30:07.093678 19573 consensus_queue.cc:1035] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:30:07.097038 19920 raft_consensus.cc:2932] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 3 LEADER]: Committing config change with OpId 3.9: config changed from index 8 to 9, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } }
I20240502 21:30:07.104002  9507 raft_consensus.cc:2932] T f0259f01acb249dd99c662151dc21916 P 19f162e164854f2da623a4bb8f9e6bf6 [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 8 to 9, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } }
I20240502 21:30:07.111443  8963 catalog_manager.cc:5484] T f0259f01acb249dd99c662151dc21916 P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: config changed from index 8 to 9, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) added. New cstate: current_term: 3 leader_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } }
I20240502 21:30:07.118080  8945 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet f0259f01acb249dd99c662151dc21916 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 21:30:07.145411 19921 log.cc:1196] T f0259f01acb249dd99c662151dc21916 P f2fdd596a8b44bbc848fe40f9f07fbe0: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/wals/f0259f01acb249dd99c662151dc21916
I20240502 21:30:07.146689  8948 catalog_manager.cc:4830] TS f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): tablet f0259f01acb249dd99c662151dc21916 (table rebalance_test_table_4 [id=1b17408dbab9492bba0ed50efd6ef43f]) successfully deleted
I20240502 21:30:07.180621 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:07.205109  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:07.205556  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:07.255487  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:30:07.268265 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:07.290069 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:07.325559 11365 raft_consensus.cc:1261] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:30:07.327342  9239 raft_consensus.cc:1261] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:30:07.328051 19686 consensus_queue.cc:1035] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:30:07.335216 19686 consensus_queue.cc:1035] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:30:07.352496  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:07.404664  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:30:07.445880 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:07.604473  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:30:07.653443  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:07.672856 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:07.674942  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:30:07.678462 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:07.722826  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:07.746841 20042 ts_tablet_manager.cc:902] T f0259f01acb249dd99c662151dc21916 P 525a093ea5bf40dc9bbf9dc39895d670: Initiating tablet copy from peer 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857)
I20240502 21:30:07.759122  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:07.763643 19458 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:30:07.763890 19458 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:30:07.764736 19458 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:30:07.767383 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
W20240502 21:30:07.768851 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:07.768996 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:30:07.769770 19458 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 21:30:07.783038  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:30:07.788424 20042 tablet_copy_client.cc:308] T f0259f01acb249dd99c662151dc21916 P 525a093ea5bf40dc9bbf9dc39895d670: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.71:35857
I20240502 21:30:07.791545 11385 tablet_copy_service.cc:140] P 4cbb5b06a6a44eb6b2145cd538b01af2: Received BeginTabletCopySession request for tablet f0259f01acb249dd99c662151dc21916 from peer 525a093ea5bf40dc9bbf9dc39895d670 ({username='slave'} at 127.27.196.66:35459)
I20240502 21:30:07.791774 11385 tablet_copy_service.cc:161] P 4cbb5b06a6a44eb6b2145cd538b01af2: Beginning new tablet copy session on tablet f0259f01acb249dd99c662151dc21916 from peer 525a093ea5bf40dc9bbf9dc39895d670 at {username='slave'} at 127.27.196.66:35459: session id = 525a093ea5bf40dc9bbf9dc39895d670-f0259f01acb249dd99c662151dc21916
I20240502 21:30:07.796348 11385 tablet_copy_source_session.cc:205] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 21:30:07.798163  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:30:07.799710 20042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0259f01acb249dd99c662151dc21916. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:30:07.806006 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:30:07.855840  8946 catalog_manager.cc:4573] TS 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4d1a31513bb64578a85adf1f622e5bc0: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:07.864670  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:30:07.872896  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:30:07.911104  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:30:07.925740 20042 tablet_copy_client.cc:783] T f0259f01acb249dd99c662151dc21916 P 525a093ea5bf40dc9bbf9dc39895d670: tablet copy: Starting download of 0 data blocks...
I20240502 21:30:07.926259 20042 tablet_copy_client.cc:647] T f0259f01acb249dd99c662151dc21916 P 525a093ea5bf40dc9bbf9dc39895d670: tablet copy: Starting download of 1 WAL segments...
W20240502 21:30:07.935909  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:30:07.939287 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:07.943409  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:07.953647 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:07.957316 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:30:07.980163  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:30:07.981014 20042 tablet_copy_client.cc:523] T f0259f01acb249dd99c662151dc21916 P 525a093ea5bf40dc9bbf9dc39895d670: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:30:07.988584 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:08.037505  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:08.059731  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:30:08.082147 20042 tablet_bootstrap.cc:492] T f0259f01acb249dd99c662151dc21916 P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap starting.
W20240502 21:30:08.191177  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:30:08.229882 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:30:08.294401 20042 tablet_bootstrap.cc:492] T f0259f01acb249dd99c662151dc21916 P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:30:08.295163 20042 tablet_bootstrap.cc:492] T f0259f01acb249dd99c662151dc21916 P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap complete.
I20240502 21:30:08.295523 20042 ts_tablet_manager.cc:1367] T f0259f01acb249dd99c662151dc21916 P 525a093ea5bf40dc9bbf9dc39895d670: Time spent bootstrapping tablet: real 0.214s	user 0.074s	sys 0.004s
I20240502 21:30:08.296587 20042 raft_consensus.cc:348] T f0259f01acb249dd99c662151dc21916 P 525a093ea5bf40dc9bbf9dc39895d670 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:30:08.296835 20042 raft_consensus.cc:729] T f0259f01acb249dd99c662151dc21916 P 525a093ea5bf40dc9bbf9dc39895d670 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Initialized, Role: FOLLOWER
I20240502 21:30:08.297135 20042 consensus_queue.cc:260] T f0259f01acb249dd99c662151dc21916 P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 3.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:30:08.298534 20042 ts_tablet_manager.cc:1398] T f0259f01acb249dd99c662151dc21916 P 525a093ea5bf40dc9bbf9dc39895d670: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 21:30:08.301827 11385 tablet_copy_service.cc:342] P 4cbb5b06a6a44eb6b2145cd538b01af2: Request end of tablet copy session 525a093ea5bf40dc9bbf9dc39895d670-f0259f01acb249dd99c662151dc21916 received from {username='slave'} at 127.27.196.66:35459
I20240502 21:30:08.302003 11385 tablet_copy_service.cc:433] P 4cbb5b06a6a44eb6b2145cd538b01af2: ending tablet copy session 525a093ea5bf40dc9bbf9dc39895d670-f0259f01acb249dd99c662151dc21916 on tablet f0259f01acb249dd99c662151dc21916 with peer 525a093ea5bf40dc9bbf9dc39895d670
I20240502 21:30:08.415378  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:08.430469 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:08.438791  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:30:08.453446 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:08.460263  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:30:08.496728 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:08.514387 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:08.535404  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:08.806983  9242 raft_consensus.cc:1203] T f0259f01acb249dd99c662151dc21916 P 525a093ea5bf40dc9bbf9dc39895d670 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.8->[3.9-3.9]   Dedup: 3.9->[]
W20240502 21:30:08.907222 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:08.953332  9377 consensus_queue.cc:237] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:30:09.005071 11365 raft_consensus.cc:1261] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:30:09.006001 19686 consensus_queue.cc:1035] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:30:09.059316 19686 raft_consensus.cc:2932] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:30:09.063699 11364 raft_consensus.cc:2932] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:30:09.071292  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:09.074452  8953 catalog_manager.cc:5484] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: config changed from index 5 to 7, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) evicted. New cstate: current_term: 2 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:30:09.110004  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:09.117681  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:30:09.127552  9221 tablet_service.cc:1514] Processing DeleteTablet for tablet 87f35aff2b104c01a9d9d48a7e9c9b23 with delete_type TABLET_DATA_TOMBSTONED (TS 525a093ea5bf40dc9bbf9dc39895d670 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44408
I20240502 21:30:09.128190  9377 consensus_queue.cc:237] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
W20240502 21:30:09.131664  9313 consensus_peers.cc:485] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467): Couldn't send request to peer e0de0545fa0547d3929111e990e6d1b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87f35aff2b104c01a9d9d48a7e9c9b23. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:30:09.133651 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:09.133813 11365 raft_consensus.cc:1261] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 21:30:09.134552 19686 consensus_queue.cc:1035] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:30:09.140031 19686 raft_consensus.cc:2932] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:30:09.146570 11364 raft_consensus.cc:2932] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:30:09.174758  8948 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 87f35aff2b104c01a9d9d48a7e9c9b23 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 21:30:09.180667  8962 catalog_manager.cc:5484] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: config changed from index 7 to 8, VOTER e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72) added. New cstate: current_term: 2 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:30:09.183470  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:30:09.201898 20237 tablet_replica.cc:330] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670: stopping tablet replica
I20240502 21:30:09.215438 20237 raft_consensus.cc:2229] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:30:09.215732 20237 raft_consensus.cc:2258] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:30:09.220757 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:09.221665 20237 ts_tablet_manager.cc:1875] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:30:09.232543  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:09.270447 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:09.280376 20237 ts_tablet_manager.cc:1888] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 21:30:09.281891 20237 log.cc:1196] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 525a093ea5bf40dc9bbf9dc39895d670: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/wals/87f35aff2b104c01a9d9d48a7e9c9b23
I20240502 21:30:09.283571  8949 catalog_manager.cc:4830] TS 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): tablet 87f35aff2b104c01a9d9d48a7e9c9b23 (table rebalance_test_table_7 [id=ef412fa8fe9249729cb99879a5c8343a]) successfully deleted
W20240502 21:30:09.420718 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:09.566524 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:09.574450  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:09.591584  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:09.593204 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:09.613780  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 21:30:09.681519  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:30:09.682494 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:09.696697 20288 ts_tablet_manager.cc:902] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2: Initiating tablet copy from peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:30:09.720116  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:09.794592 20288 tablet_copy_client.cc:308] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:43203
I20240502 21:30:09.795657  9399 tablet_copy_service.cc:140] P f2fdd596a8b44bbc848fe40f9f07fbe0: Received BeginTabletCopySession request for tablet 87f35aff2b104c01a9d9d48a7e9c9b23 from peer e0de0545fa0547d3929111e990e6d1b2 ({username='slave'} at 127.27.196.72:46235)
I20240502 21:30:09.795872  9399 tablet_copy_service.cc:161] P f2fdd596a8b44bbc848fe40f9f07fbe0: Beginning new tablet copy session on tablet 87f35aff2b104c01a9d9d48a7e9c9b23 from peer e0de0545fa0547d3929111e990e6d1b2 at {username='slave'} at 127.27.196.72:46235: session id = e0de0545fa0547d3929111e990e6d1b2-87f35aff2b104c01a9d9d48a7e9c9b23
I20240502 21:30:09.799685  9399 tablet_copy_source_session.cc:205] T 87f35aff2b104c01a9d9d48a7e9c9b23 P f2fdd596a8b44bbc848fe40f9f07fbe0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:30:09.802276 20288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87f35aff2b104c01a9d9d48a7e9c9b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:30:09.820267 20288 tablet_copy_client.cc:783] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Starting download of 0 data blocks...
I20240502 21:30:09.820683 20288 tablet_copy_client.cc:647] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:30:09.841645 20288 tablet_copy_client.cc:523] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:30:09.865610 20288 tablet_bootstrap.cc:492] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2: Bootstrap starting.
I20240502 21:30:09.892076 11546 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240502 21:29:07.802417 11553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:07.802904 11553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:07.834414 11553 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20240502 21:29:08.553372 11599 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.27.196.126:34625, user_credentials={real_user=slave}} blocked reactor thread for 45508.1us
I20240502 21:29:11.327946 11553 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 21:29:13.614727 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a237d7bda970485480e98020c6ac738f: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:13.615096 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 445d2e6ec7154b80abec5fff346b8f5e: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:13.615374 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f52c6c3804274777abfa00bb33ae9b16: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:13.615516 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c0a0ef39a2a246449eb582cc1d993484: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:13.615713 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e8eef05f850243d0bdc7e1899f8c6139: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:13.615873 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 265d5656d7c048aa9795a906661b8ea8: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:13.616017 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16e8100140034581b2a773dcf740998d: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:13.616147 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a237d7bda970485480e98020c6ac738f: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:13.616310 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1b17408dbab9492bba0ed50efd6ef43f: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:13.616462 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5802da0404e74a07bca290cdc59b279b: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:13.616627 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef412fa8fe9249729cb99879a5c8343a: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:13.616767 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f52c6c3804274777abfa00bb33ae9b16: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:13.616878 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e8eef05f850243d0bdc7e1899f8c6139: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:13.617022 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c0a0ef39a2a246449eb582cc1d993484: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:13.617157 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 265d5656d7c048aa9795a906661b8ea8: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server 4cbb5b06a6a44eb6b2145cd538b01af2
I20240502 21:29:14.295392 11553 rebalancer_tool.cc:1074] tablet 4d1a31513bb64578a85adf1f622e5bc0: '525a093ea5bf40dc9bbf9dc39895d670' -> '6ef0beaf2f6941afb6ea9f9f274517fe' move scheduled
I20240502 21:29:14.993587 11553 rebalancer_tool.cc:1074] tablet 51936fb1f678458a8795f8246590addb: '19f162e164854f2da623a4bb8f9e6bf6' -> 'd10e9e156ba74a97a221c85b68ab8bc4' move scheduled
I20240502 21:29:15.606400 11553 rebalancer_tool.cc:1074] tablet 8b4486f03fd9474aa0bb2cb4c9149437: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '4cbb5b06a6a44eb6b2145cd538b01af2' move scheduled
I20240502 21:29:16.277922 11553 rebalancer_tool.cc:1074] tablet a2f3edaf1c874cefb820f027f90d1185: '5543ddab5b484642b139b7c848932a58' -> 'e0de0545fa0547d3929111e990e6d1b2' move scheduled
I20240502 21:29:16.930599 11553 rebalancer_tool.cc:1074] tablet 9240163cfac446d2a14dbd9fcbbf9ff7: '525a093ea5bf40dc9bbf9dc39895d670' -> '6ef0beaf2f6941afb6ea9f9f274517fe' move scheduled
I20240502 21:29:17.600886 11553 rebalancer_tool.cc:1074] tablet 3caa36212ba444aa88d153ff9f90122e: '19f162e164854f2da623a4bb8f9e6bf6' -> 'd10e9e156ba74a97a221c85b68ab8bc4' move scheduled
I20240502 21:29:18.680464 11553 rebalancer_tool.cc:1074] tablet 07d693b680a54e12871121737e3da744: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '4cbb5b06a6a44eb6b2145cd538b01af2' move scheduled
I20240502 21:29:19.258936 11553 rebalancer_tool.cc:1074] tablet f9ae8fac693d4d68b6741f2509e1265e: '5543ddab5b484642b139b7c848932a58' -> 'e0de0545fa0547d3929111e990e6d1b2' move scheduled
I20240502 21:29:19.361639 11553 rebalancer_tool.cc:1084] tablet 8a2e06253c51405fa9b6bc9e87844760: '525a093ea5bf40dc9bbf9dc39895d670' -> 'd10e9e156ba74a97a221c85b68ab8bc4' move ignored: Illegal state: tablet 8a2e06253c51405fa9b6bc9e87844760: replica d10e9e156ba74a97a221c85b68ab8bc4 is already present
I20240502 21:29:19.361946 11553 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 21:29:20.449687 13306 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.27.196.126:34625, user_credentials={real_user=slave}} blocked reactor thread for 68756.9us
W20240502 21:29:21.466135 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e8eef05f850243d0bdc7e1899f8c6139: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:21.466312 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 445d2e6ec7154b80abec5fff346b8f5e: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:21.466415 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a237d7bda970485480e98020c6ac738f: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:21.466571 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1b17408dbab9492bba0ed50efd6ef43f: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:21.466651 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 265d5656d7c048aa9795a906661b8ea8: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:21.466873 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f52c6c3804274777abfa00bb33ae9b16: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:21.467003 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16e8100140034581b2a773dcf740998d: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:21.467092 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e8eef05f850243d0bdc7e1899f8c6139: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:21.468454 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a237d7bda970485480e98020c6ac738f: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:21.468605 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c0a0ef39a2a246449eb582cc1d993484: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:21.468719 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 265d5656d7c048aa9795a906661b8ea8: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:21.468813 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5802da0404e74a07bca290cdc59b279b: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:21.468945 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef412fa8fe9249729cb99879a5c8343a: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:29:21.855041 11553 rebalancer_tool.cc:1074] tablet aeee59c5512d448d8f986f959dce1cc8: '5543ddab5b484642b139b7c848932a58' -> '' move scheduled
I20240502 21:29:22.055068 11553 rebalancer_tool.cc:1074] tablet 58c7004ceb044f5fa9dec54910046b69: '6ef0beaf2f6941afb6ea9f9f274517fe' -> '' move scheduled
I20240502 21:29:22.126663 11553 rebalancer_tool.cc:1074] tablet 8a2e06253c51405fa9b6bc9e87844760: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move scheduled
I20240502 21:29:23.028993 11553 rebalancer_tool.cc:1074] tablet e46065d94e874293a836bc74241b8e06: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> 'd10e9e156ba74a97a221c85b68ab8bc4' move scheduled
I20240502 21:29:23.226168 11553 rebalancer_tool.cc:1084] tablet 94d2cdf8ae6941668b5f680c558656e8: '19f162e164854f2da623a4bb8f9e6bf6' -> 'e0de0545fa0547d3929111e990e6d1b2' move ignored: Illegal state: tablet 94d2cdf8ae6941668b5f680c558656e8: replica e0de0545fa0547d3929111e990e6d1b2 is already present
I20240502 21:29:23.226334 11553 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 21:29:25.547899 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f52c6c3804274777abfa00bb33ae9b16: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:25.548943 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c0a0ef39a2a246449eb582cc1d993484: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:25.549124 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5802da0404e74a07bca290cdc59b279b: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:25.549232 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1b17408dbab9492bba0ed50efd6ef43f: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:25.549319 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16e8100140034581b2a773dcf740998d: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:25.549464 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f52c6c3804274777abfa00bb33ae9b16: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:25.549552 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 445d2e6ec7154b80abec5fff346b8f5e: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:25.549743 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e8eef05f850243d0bdc7e1899f8c6139: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:25.549846 11553 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 265d5656d7c048aa9795a906661b8ea8: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server 6ef0beaf2f6941afb6ea9f9f274517fe
I20240502 21:29:25.703612 11553 rebalancer_tool.cc:1074] tablet 94d2cdf8ae6941668b5f680c558656e8: '19f162e164854f2da623a4bb8f9e6bf6' -> '' move scheduled
I20240502 21:29:25.957688 11553 rebalancer_tool.cc:1074] tablet cffcb4f7f7754262beeb06168097f6dc: '5543ddab5b484642b139b7c848932a58' -> '' move scheduled
I20240502 21:29:35.567387 11553 rebalancer_tool.cc:1136] tablet aeee59c5512d448d8f986f959dce1cc8: '5543ddab5b484642b139b7c848932a58' -> '' move completed: OK
I20240502 21:29:51.617880 11553 rebalancer_tool.cc:1123] tablet 4d1a31513bb64578a85adf1f622e5bc0: '525a093ea5bf40dc9bbf9dc39895d670' -> '6ef0beaf2f6941afb6ea9f9f274517fe' move is abandoned: Not found: Server with UUID 525a093ea5bf40dc9bbf9dc39895d670 not a member of the config. RaftConfig: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:29:54.737628 11553 rebalancer_tool.cc:1136] tablet 8b4486f03fd9474aa0bb2cb4c9149437: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '4cbb5b06a6a44eb6b2145cd538b01af2' move completed: OK
I20240502 21:29:57.710150 11553 rebalancer_tool.cc:1136] tablet a2f3edaf1c874cefb820f027f90d1185: '5543ddab5b484642b139b7c848932a58' -> 'e0de0545fa0547d3929111e990e6d1b2' move completed: OK
I20240502 21:29:57.890606 11553 rebalancer_tool.cc:1136] tablet 3caa36212ba444aa88d153ff9f90122e: '19f162e164854f2da623a4bb8f9e6bf6' -> 'd10e9e156ba74a97a221c85b68ab8bc4' move completed: OK
I20240502 21:29:58.167903 11553 rebalancer_tool.cc:1074] tablet be34ec89876d42b69efe1878c0d9bc19: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '' move scheduled
I20240502 21:29:58.275127 11553 rebalancer_tool.cc:1136] tablet be34ec89876d42b69efe1878c0d9bc19: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '' move completed: OK
I20240502 21:29:58.346753 11553 rebalancer_tool.cc:1136] tablet 94d2cdf8ae6941668b5f680c558656e8: '19f162e164854f2da623a4bb8f9e6bf6' -> '' move completed: OK
I20240502 21:29:58.432837 11553 rebalancer_tool.cc:1136] tablet 58c7004ceb044f5fa9dec54910046b69: '6ef0beaf2f6941afb6ea9f9f274517fe' -> '' move completed: OK
I20240502 21:29:58.586376 11553 rebalancer_tool.cc:1136] tablet 8a2e06253c51405fa9b6bc9e87844760: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move completed: OK
I20240502 21:29:58.737031 11553 rebalancer_tool.cc:1136] tablet e46065d94e874293a836bc74241b8e06: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> 'd10e9e156ba74a97a221c85b68ab8bc4' move completed: OK
I20240502 21:29:59.055341 11553 rebalancer_tool.cc:1136] tablet f9ae8fac693d4d68b6741f2509e1265e: '5543ddab5b484642b139b7c848932a58' -> 'e0de0545fa0547d3929111e990e6d1b2' move completed: OK
I20240502 21:29:59.282581 11553 rebalancer_tool.cc:1136] tablet cffcb4f7f7754262beeb06168097f6dc: '5543ddab5b484642b139b7c848932a58' -> '' move completed: OK
I20240502 21:29:59.443372 11553 rebalancer_tool.cc:1136] tablet 51936fb1f678458a8795f8246590addb: '19f162e164854f2da623a4bb8f9e6bf6' -> 'd10e9e156ba74a97a221c85b68ab8bc4' move completed: OK
I20240502 21:29:59.527498 11553 rebalancer_tool.cc:1136] tablet 07d693b680a54e12871121737e3da744: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '4cbb5b06a6a44eb6b2145cd538b01af2' move completed: OK
I20240502 21:29:59.600868 11553 rebalancer_tool.cc:1136] tablet 9240163cfac446d2a14dbd9fcbbf9ff7: '525a093ea5bf40dc9bbf9dc39895d670' -> '6ef0beaf2f6941afb6ea9f9f274517fe' move completed: OK
I20240502 21:29:59.668227 11553 rebalancer_tool.cc:1074] tablet 41ee481a3ece47a0905e93c66dbf949d: '5543ddab5b484642b139b7c848932a58' -> '' move scheduled
I20240502 21:29:59.790599 11553 rebalancer_tool.cc:1074] tablet 32c306d724954a16be21a34aa4cc0b5f: '19f162e164854f2da623a4bb8f9e6bf6' -> '' move scheduled
I20240502 21:29:59.835580 11553 rebalancer_tool.cc:1074] tablet 1497b781f36c43b39d22a05bb524c59d: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '' move scheduled
I20240502 21:30:00.119755 11553 rebalancer_tool.cc:1074] tablet 3d1a5050d3f944969a560994a23700ad: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move scheduled
I20240502 21:30:00.191159 11553 rebalancer_tool.cc:1074] tablet 87f35aff2b104c01a9d9d48a7e9c9b23: '19f162e164854f2da623a4bb8f9e6bf6' -> '' move scheduled
I20240502 21:30:00.244009 11553 rebalancer_tool.cc:1136] tablet 87f35aff2b104c01a9d9d48a7e9c9b23: '19f162e164854f2da623a4bb8f9e6bf6' -> '' move completed: OK
I20240502 21:30:00.338927 11553 rebalancer_tool.cc:1136] tablet 3d1a5050d3f944969a560994a23700ad: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move completed: OK
I20240502 21:30:00.729887 11553 rebalancer_tool.cc:1136] tablet 1497b781f36c43b39d22a05bb524c59d: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '' move completed: OK
I20240502 21:30:00.877110 11553 rebalancer_tool.cc:1136] tablet 32c306d724954a16be21a34aa4cc0b5f: '19f162e164854f2da623a4bb8f9e6bf6' -> '' move completed: OK
I20240502 21:30:01.083914 11553 rebalancer_tool.cc:1136] tablet 41ee481a3ece47a0905e93c66dbf949d: '5543ddab5b484642b139b7c848932a58' -> '' move completed: OK
I20240502 21:30:01.224750 11553 rebalancer_tool.cc:1074] tablet cdf72d25f0f847e5998acbc53979e64d: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move scheduled
I20240502 21:30:01.359843 11553 rebalancer_tool.cc:1074] tablet a7c6b881c9ef4cdca787f065778e526f: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move scheduled
I20240502 21:30:07.074774 11553 rebalancer_tool.cc:1136] tablet a7c6b881c9ef4cdca787f065778e526f: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move completed: OK
I20240502 21:30:07.226356 11553 rebalancer_tool.cc:1136] tablet cdf72d25f0f847e5998acbc53979e64d: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move completed: OK
Illegal state: tablet server 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): unacceptable health status UNAVAILABLE

W20240502 21:30:09.944034  9175 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.72:40467, user_credentials={real_user=slave}} blocked reactor thread for 44693.7us
W20240502 21:30:09.959061 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:09.992703  8946 catalog_manager.cc:4573] TS 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4d1a31513bb64578a85adf1f622e5bc0: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:10.062862 20288 tablet_bootstrap.cc:492] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:30:10.064886 20288 tablet_bootstrap.cc:492] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2: Bootstrap complete.
I20240502 21:30:10.065910 20288 ts_tablet_manager.cc:1367] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2: Time spent bootstrapping tablet: real 0.200s	user 0.057s	sys 0.005s
I20240502 21:30:10.094434 20288 raft_consensus.cc:348] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:30:10.095665 20288 raft_consensus.cc:729] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0de0545fa0547d3929111e990e6d1b2, State: Initialized, Role: FOLLOWER
I20240502 21:30:10.096117 20288 consensus_queue.cc:260] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:30:10.097183 20288 ts_tablet_manager.cc:1398] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 21:30:10.098517  9399 tablet_copy_service.cc:342] P f2fdd596a8b44bbc848fe40f9f07fbe0: Request end of tablet copy session e0de0545fa0547d3929111e990e6d1b2-87f35aff2b104c01a9d9d48a7e9c9b23 received from {username='slave'} at 127.27.196.72:46235
I20240502 21:30:10.098646  9399 tablet_copy_service.cc:433] P f2fdd596a8b44bbc848fe40f9f07fbe0: ending tablet copy session e0de0545fa0547d3929111e990e6d1b2-87f35aff2b104c01a9d9d48a7e9c9b23 on tablet 87f35aff2b104c01a9d9d48a7e9c9b23 with peer e0de0545fa0547d3929111e990e6d1b2
I20240502 21:30:10.105027 11496 raft_consensus.cc:1203] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8]   Dedup: 2.8->[]
W20240502 21:30:10.180388  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:30:10.248761  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:30:10.383795  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:30:10.384979  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:30:10.414662 11547 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240502 21:29:08.200567 11555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:08.201272 11555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:08.220414 11555 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20240502 21:29:10.047780 11629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.27.196.66:39071, user_credentials={real_user=slave}} blocked reactor thread for 63616.1us
I20240502 21:29:12.786618 11555 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 21:29:15.090342 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f52c6c3804274777abfa00bb33ae9b16: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:15.096577 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c0a0ef39a2a246449eb582cc1d993484: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:15.098702 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a237d7bda970485480e98020c6ac738f: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:15.099088 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 445d2e6ec7154b80abec5fff346b8f5e: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:15.099234 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1b17408dbab9492bba0ed50efd6ef43f: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:15.099450 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f52c6c3804274777abfa00bb33ae9b16: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:15.099609 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e8eef05f850243d0bdc7e1899f8c6139: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:15.099720 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 265d5656d7c048aa9795a906661b8ea8: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:15.099802 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16e8100140034581b2a773dcf740998d: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:15.111006 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a237d7bda970485480e98020c6ac738f: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:15.111416 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5802da0404e74a07bca290cdc59b279b: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:15.111577 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef412fa8fe9249729cb99879a5c8343a: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:15.111681 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f52c6c3804274777abfa00bb33ae9b16: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:15.111809 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c0a0ef39a2a246449eb582cc1d993484: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:15.111897 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e8eef05f850243d0bdc7e1899f8c6139: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:15.112007 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 265d5656d7c048aa9795a906661b8ea8: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server e0de0545fa0547d3929111e990e6d1b2
I20240502 21:29:15.724243 11555 rebalancer_tool.cc:1074] tablet 58c7004ceb044f5fa9dec54910046b69: '525a093ea5bf40dc9bbf9dc39895d670' -> '6ef0beaf2f6941afb6ea9f9f274517fe' move scheduled
I20240502 21:29:16.345896 11555 rebalancer_tool.cc:1074] tablet cdf72d25f0f847e5998acbc53979e64d: '19f162e164854f2da623a4bb8f9e6bf6' -> '4cbb5b06a6a44eb6b2145cd538b01af2' move scheduled
I20240502 21:29:16.811244 11555 rebalancer_tool.cc:1074] tablet 8b4486f03fd9474aa0bb2cb4c9149437: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> 'd10e9e156ba74a97a221c85b68ab8bc4' move scheduled
I20240502 21:29:17.377306 11555 rebalancer_tool.cc:1074] tablet cffcb4f7f7754262beeb06168097f6dc: '5543ddab5b484642b139b7c848932a58' -> 'e0de0545fa0547d3929111e990e6d1b2' move scheduled
I20240502 21:29:18.326193 11555 rebalancer_tool.cc:1074] tablet 3d1a5050d3f944969a560994a23700ad: '525a093ea5bf40dc9bbf9dc39895d670' -> '6ef0beaf2f6941afb6ea9f9f274517fe' move scheduled
I20240502 21:29:19.100623 11555 rebalancer_tool.cc:1074] tablet 8129d13163f64368b731ba4eaeaaa81c: '19f162e164854f2da623a4bb8f9e6bf6' -> '4cbb5b06a6a44eb6b2145cd538b01af2' move scheduled
I20240502 21:29:19.489076 11555 rebalancer_tool.cc:1074] tablet be34ec89876d42b69efe1878c0d9bc19: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> 'd10e9e156ba74a97a221c85b68ab8bc4' move scheduled
I20240502 21:29:19.569252 11555 rebalancer_tool.cc:1084] tablet f9ae8fac693d4d68b6741f2509e1265e: '5543ddab5b484642b139b7c848932a58' -> 'e0de0545fa0547d3929111e990e6d1b2' move ignored: Illegal state: tablet f9ae8fac693d4d68b6741f2509e1265e: replica e0de0545fa0547d3929111e990e6d1b2 is already present
I20240502 21:29:19.569485 11555 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 21:29:21.823938 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e8eef05f850243d0bdc7e1899f8c6139: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:21.824584 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c0a0ef39a2a246449eb582cc1d993484: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:21.824843 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f52c6c3804274777abfa00bb33ae9b16: could not find any suitable replica to move from server 6ef0beaf2f6941afb6ea9f9f274517fe to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:21.825080 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 265d5656d7c048aa9795a906661b8ea8: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:21.825243 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1b17408dbab9492bba0ed50efd6ef43f: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:21.825443 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e8eef05f850243d0bdc7e1899f8c6139: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:21.825616 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a237d7bda970485480e98020c6ac738f: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:21.825862 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16e8100140034581b2a773dcf740998d: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:21.826220 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 265d5656d7c048aa9795a906661b8ea8: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:21.826372 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1b17408dbab9492bba0ed50efd6ef43f: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server 4cbb5b06a6a44eb6b2145cd538b01af2
I20240502 21:29:21.925822 11555 rebalancer_tool.cc:1074] tablet 51936fb1f678458a8795f8246590addb: '5543ddab5b484642b139b7c848932a58' -> '' move scheduled
I20240502 21:29:22.352079 11555 rebalancer_tool.cc:1074] tablet 94d2cdf8ae6941668b5f680c558656e8: '525a093ea5bf40dc9bbf9dc39895d670' -> 'e0de0545fa0547d3929111e990e6d1b2' move scheduled
I20240502 21:29:22.493507 11555 rebalancer_tool.cc:1074] tablet 4d1a31513bb64578a85adf1f622e5bc0: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '' move scheduled
W20240502 21:29:23.150882 13732 thread.cc:636] client-negotiator [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.109s	sys 0.202s
W20240502 21:29:23.152271 13732 thread.cc:602] client-negotiator [worker] (thread pool) Time spent starting thread: real 0.515s	user 0.109s	sys 0.203s
I20240502 21:29:23.716408 11555 rebalancer_tool.cc:1074] tablet c56ac926025e47218e9df4088184a278: '5543ddab5b484642b139b7c848932a58' -> '6ef0beaf2f6941afb6ea9f9f274517fe' move scheduled
I20240502 21:29:24.659132 11555 rebalancer_tool.cc:1074] tablet 79dedaa327b6479e8325f91481c1688e: '19f162e164854f2da623a4bb8f9e6bf6' -> 'd10e9e156ba74a97a221c85b68ab8bc4' move scheduled
I20240502 21:29:24.943331 11555 rebalancer_tool.cc:1074] tablet 1497b781f36c43b39d22a05bb524c59d: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move scheduled
I20240502 21:29:25.082424 11555 rebalancer_tool.cc:1084] tablet e46065d94e874293a836bc74241b8e06: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> 'd10e9e156ba74a97a221c85b68ab8bc4' move ignored: Illegal state: tablet e46065d94e874293a836bc74241b8e06: replica d10e9e156ba74a97a221c85b68ab8bc4 is already present
I20240502 21:29:25.082558 11555 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 21:29:26.995082 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16e8100140034581b2a773dcf740998d: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:26.995290 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c0a0ef39a2a246449eb582cc1d993484: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:26.995657 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a237d7bda970485480e98020c6ac738f: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:26.995838 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef412fa8fe9249729cb99879a5c8343a: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:26.995999 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f52c6c3804274777abfa00bb33ae9b16: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:27.006744 11555 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 445d2e6ec7154b80abec5fff346b8f5e: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server 6ef0beaf2f6941afb6ea9f9f274517fe
I20240502 21:29:27.152771 11555 rebalancer_tool.cc:1074] tablet a7c6b881c9ef4cdca787f065778e526f: '5543ddab5b484642b139b7c848932a58' -> '' move scheduled
I20240502 21:29:27.362159 11555 rebalancer_tool.cc:1074] tablet 07d693b680a54e12871121737e3da744: '6ef0beaf2f6941afb6ea9f9f274517fe' -> '' move scheduled
I20240502 21:29:59.095218 11555 rebalancer_tool.cc:1136] tablet 8b4486f03fd9474aa0bb2cb4c9149437: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> 'd10e9e156ba74a97a221c85b68ab8bc4' move completed: OK
I20240502 21:29:59.367825 11555 rebalancer_tool.cc:1136] tablet 51936fb1f678458a8795f8246590addb: '5543ddab5b484642b139b7c848932a58' -> '' move completed: OK
I20240502 21:29:59.463593 11555 rebalancer_tool.cc:1136] tablet cffcb4f7f7754262beeb06168097f6dc: '5543ddab5b484642b139b7c848932a58' -> 'e0de0545fa0547d3929111e990e6d1b2' move completed: OK
I20240502 21:29:59.545394 11555 rebalancer_tool.cc:1136] tablet 3d1a5050d3f944969a560994a23700ad: '525a093ea5bf40dc9bbf9dc39895d670' -> '6ef0beaf2f6941afb6ea9f9f274517fe' move completed: OK
I20240502 21:29:59.635020 11555 rebalancer_tool.cc:1136] tablet 58c7004ceb044f5fa9dec54910046b69: '525a093ea5bf40dc9bbf9dc39895d670' -> '6ef0beaf2f6941afb6ea9f9f274517fe' move completed: Incomplete: tablet 58c7004ceb044f5fa9dec54910046b69, TS 525a093ea5bf40dc9bbf9dc39895d670 -> TS 6ef0beaf2f6941afb6ea9f9f274517fe move failed, target replica disappeared
I20240502 21:30:01.613911 11555 rebalancer_tool.cc:1074] tablet f0259f01acb249dd99c662151dc21916: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '' move scheduled
I20240502 21:30:01.868624 11555 rebalancer_tool.cc:1136] tablet 1497b781f36c43b39d22a05bb524c59d: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move completed: OK
I20240502 21:30:02.153642 11555 rebalancer_tool.cc:1136] tablet a7c6b881c9ef4cdca787f065778e526f: '5543ddab5b484642b139b7c848932a58' -> '' move completed: OK
I20240502 21:30:02.319506 11555 rebalancer_tool.cc:1136] tablet 79dedaa327b6479e8325f91481c1688e: '19f162e164854f2da623a4bb8f9e6bf6' -> 'd10e9e156ba74a97a221c85b68ab8bc4' move completed: OK
I20240502 21:30:02.373715 11555 rebalancer_tool.cc:1136] tablet c56ac926025e47218e9df4088184a278: '5543ddab5b484642b139b7c848932a58' -> '6ef0beaf2f6941afb6ea9f9f274517fe' move completed: OK
I20240502 21:30:02.527766 11555 rebalancer_tool.cc:1136] tablet 4d1a31513bb64578a85adf1f622e5bc0: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '' move completed: OK
I20240502 21:30:02.636456 11555 rebalancer_tool.cc:1136] tablet 94d2cdf8ae6941668b5f680c558656e8: '525a093ea5bf40dc9bbf9dc39895d670' -> 'e0de0545fa0547d3929111e990e6d1b2' move completed: OK
I20240502 21:30:07.175468 11555 rebalancer_tool.cc:1136] tablet be34ec89876d42b69efe1878c0d9bc19: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> 'd10e9e156ba74a97a221c85b68ab8bc4' move completed: OK
I20240502 21:30:07.428521 11555 rebalancer_tool.cc:1136] tablet cdf72d25f0f847e5998acbc53979e64d: '19f162e164854f2da623a4bb8f9e6bf6' -> '4cbb5b06a6a44eb6b2145cd538b01af2' move completed: OK
I20240502 21:30:07.490319 11555 rebalancer_tool.cc:1136] tablet 07d693b680a54e12871121737e3da744: '6ef0beaf2f6941afb6ea9f9f274517fe' -> '' move completed: OK
I20240502 21:30:07.589592 11555 rebalancer_tool.cc:1136] tablet 8129d13163f64368b731ba4eaeaaa81c: '19f162e164854f2da623a4bb8f9e6bf6' -> '4cbb5b06a6a44eb6b2145cd538b01af2' move completed: OK
I20240502 21:30:07.897394 11555 rebalancer_tool.cc:1074] tablet de3287296c4547688388388dd4c7e638: '5543ddab5b484642b139b7c848932a58' -> '' move scheduled
I20240502 21:30:08.076279 11555 rebalancer_tool.cc:1084] tablet f82c35077224414cb284472d49d7de4f: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move ignored: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:08.076575 11555 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): unacceptable health status UNAVAILABLE

I20240502 21:30:10.420598  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:10.422024  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 21:30:10.423823  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:30:10.424823  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:10.427945 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:30:10.446265  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240502 21:30:10.447484  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:30:10.454790  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:30:10.499540  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:10.521095  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:30:10.526499 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:30:10.558010 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240502 21:30:10.559639 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:10.559859 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:10.568500  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:10.597566  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:30:10.659511 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:30:10.709626  9105 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "de3287296c4547688388388dd4c7e638"
dest_uuid: "5543ddab5b484642b139b7c848932a58"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:42516
I20240502 21:30:10.709892  9105 raft_consensus.cc:595] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:30:10.878052  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:30:10.886301  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:30:10.899150  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:10.900924 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:10.950775  9085 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:30:10.951154 19458 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:30:10.951483 19458 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:30:10.952486 19458 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:30:10.958974 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
I20240502 21:30:10.960171  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:10.960372 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:10.960477 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:30:10.966897 19458 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 21:30:10.984067 11073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:10.985641 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:10.998584 20245 raft_consensus.cc:979] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58: : Instructing follower d10e9e156ba74a97a221c85b68ab8bc4 to start an election
I20240502 21:30:10.998773 20245 raft_consensus.cc:1067] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Signalling peer d10e9e156ba74a97a221c85b68ab8bc4 to start an election
I20240502 21:30:11.000280 11093 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "de3287296c4547688388388dd4c7e638"
dest_uuid: "d10e9e156ba74a97a221c85b68ab8bc4"
 from {username='slave'} at 127.27.196.65:42255
I20240502 21:30:11.000561 11093 raft_consensus.cc:482] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:30:11.000658 11093 raft_consensus.cc:3037] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:30:11.004313 11093 raft_consensus.cc:504] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:30:11.011030  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:11.011543 11093 leader_election.cc:288] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 election: Requested vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:30:11.030742  9105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de3287296c4547688388388dd4c7e638" candidate_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "5543ddab5b484642b139b7c848932a58"
I20240502 21:30:11.031101  9105 raft_consensus.cc:3032] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:30:11.031203  9105 raft_consensus.cc:729] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Running, Role: LEADER
I20240502 21:30:11.031610  9105 consensus_queue.cc:260] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:30:11.032106  9105 raft_consensus.cc:3037] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:30:11.034943  9105 raft_consensus.cc:2454] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d10e9e156ba74a97a221c85b68ab8bc4 in term 2.
W20240502 21:30:11.038074 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:11.039420 11025 leader_election.cc:302] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5543ddab5b484642b139b7c848932a58, d10e9e156ba74a97a221c85b68ab8bc4; no voters: 
I20240502 21:30:11.039921 19978 raft_consensus.cc:2781] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:30:11.076259  9507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de3287296c4547688388388dd4c7e638" candidate_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6"
I20240502 21:30:11.076622  9507 raft_consensus.cc:3037] T de3287296c4547688388388dd4c7e638 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:30:11.076829 19978 raft_consensus.cc:686] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 LEADER]: Becoming Leader. State: Replica: d10e9e156ba74a97a221c85b68ab8bc4, State: Running, Role: LEADER
I20240502 21:30:11.078192 19978 consensus_queue.cc:237] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:30:11.081302  9507 raft_consensus.cc:2454] T de3287296c4547688388388dd4c7e638 P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d10e9e156ba74a97a221c85b68ab8bc4 in term 2.
I20240502 21:30:11.099545  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:11.116672 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:11.130695  8958 catalog_manager.cc:5484] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4 reported cstate change: term changed from 1 to 2, leader changed from 5543ddab5b484642b139b7c848932a58 (127.27.196.65) to d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69). New cstate: current_term: 2 leader_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:30:11.136946 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:11.147537 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:11.170924  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:11.175798  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:11.437074  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:11.458997  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:11.461349  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:11.462337 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:11.470755 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:11.512598 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:11.526458  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:11.563843  9507 raft_consensus.cc:1261] T de3287296c4547688388388dd4c7e638 P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Refusing update from remote peer d10e9e156ba74a97a221c85b68ab8bc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:30:11.565073 19978 consensus_queue.cc:1035] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 21:30:11.641968  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:30:11.664535  9105 raft_consensus.cc:1261] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Refusing update from remote peer d10e9e156ba74a97a221c85b68ab8bc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:30:11.670120 19978 consensus_queue.cc:1035] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 21:30:11.687281 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:11.963799 11550 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240502 21:29:08.286904 11554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:08.294759 11554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:08.326277 11554 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:29:12.352319 11554 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 21:29:14.084126 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e8eef05f850243d0bdc7e1899f8c6139: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:14.084393 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16e8100140034581b2a773dcf740998d: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:14.084509 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 445d2e6ec7154b80abec5fff346b8f5e: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:14.084676 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a237d7bda970485480e98020c6ac738f: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:14.084863 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f52c6c3804274777abfa00bb33ae9b16: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:14.084987 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c0a0ef39a2a246449eb582cc1d993484: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:14.085139 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 265d5656d7c048aa9795a906661b8ea8: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:14.085307 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1b17408dbab9492bba0ed50efd6ef43f: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:14.085448 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5802da0404e74a07bca290cdc59b279b: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:14.085564 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef412fa8fe9249729cb99879a5c8343a: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:14.085659 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f52c6c3804274777abfa00bb33ae9b16: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:14.085755 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16e8100140034581b2a773dcf740998d: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:14.085842 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c0a0ef39a2a246449eb582cc1d993484: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:14.085975 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e8eef05f850243d0bdc7e1899f8c6139: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server 4cbb5b06a6a44eb6b2145cd538b01af2
I20240502 21:29:15.088482 11554 rebalancer_tool.cc:1074] tablet 07d693b680a54e12871121737e3da744: '525a093ea5bf40dc9bbf9dc39895d670' -> '6ef0beaf2f6941afb6ea9f9f274517fe' move scheduled
I20240502 21:29:15.766113 11554 rebalancer_tool.cc:1074] tablet aeee59c5512d448d8f986f959dce1cc8: '19f162e164854f2da623a4bb8f9e6bf6' -> 'd10e9e156ba74a97a221c85b68ab8bc4' move scheduled
I20240502 21:29:16.550683 11554 rebalancer_tool.cc:1074] tablet 41ee481a3ece47a0905e93c66dbf949d: '5543ddab5b484642b139b7c848932a58' -> '4cbb5b06a6a44eb6b2145cd538b01af2' move scheduled
I20240502 21:29:17.209692 11554 rebalancer_tool.cc:1074] tablet 1497b781f36c43b39d22a05bb524c59d: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> 'e0de0545fa0547d3929111e990e6d1b2' move scheduled
I20240502 21:29:17.678491 11554 rebalancer_tool.cc:1074] tablet 32c306d724954a16be21a34aa4cc0b5f: '525a093ea5bf40dc9bbf9dc39895d670' -> '6ef0beaf2f6941afb6ea9f9f274517fe' move scheduled
I20240502 21:29:18.133153 11554 rebalancer_tool.cc:1074] tablet 8a2e06253c51405fa9b6bc9e87844760: '19f162e164854f2da623a4bb8f9e6bf6' -> 'd10e9e156ba74a97a221c85b68ab8bc4' move scheduled
I20240502 21:29:19.056140 11554 rebalancer_tool.cc:1074] tablet a2f3edaf1c874cefb820f027f90d1185: '5543ddab5b484642b139b7c848932a58' -> '4cbb5b06a6a44eb6b2145cd538b01af2' move scheduled
I20240502 21:29:19.436527 11554 rebalancer_tool.cc:1084] tablet f82c35077224414cb284472d49d7de4f: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> 'e0de0545fa0547d3929111e990e6d1b2' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }.
  Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } }
I20240502 21:29:19.436771 11554 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 21:29:21.524076 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 265d5656d7c048aa9795a906661b8ea8: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:21.524268 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e8eef05f850243d0bdc7e1899f8c6139: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:21.524703 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e8eef05f850243d0bdc7e1899f8c6139: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:21.531006 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef412fa8fe9249729cb99879a5c8343a: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:21.531217 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 265d5656d7c048aa9795a906661b8ea8: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:21.531458 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f52c6c3804274777abfa00bb33ae9b16: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:21.531615 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1b17408dbab9492bba0ed50efd6ef43f: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:21.531725 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c0a0ef39a2a246449eb582cc1d993484: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:21.538259 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e8eef05f850243d0bdc7e1899f8c6139: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:21.538441 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5802da0404e74a07bca290cdc59b279b: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server e0de0545fa0547d3929111e990e6d1b2
I20240502 21:29:21.625890 11554 rebalancer_tool.cc:1074] tablet 4d1a31513bb64578a85adf1f622e5bc0: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '' move scheduled
I20240502 21:29:22.119778 11554 rebalancer_tool.cc:1074] tablet 79dedaa327b6479e8325f91481c1688e: '19f162e164854f2da623a4bb8f9e6bf6' -> 'e0de0545fa0547d3929111e990e6d1b2' move scheduled
I20240502 21:29:22.209829 11554 rebalancer_tool.cc:1074] tablet 51936fb1f678458a8795f8246590addb: '5543ddab5b484642b139b7c848932a58' -> '' move scheduled
I20240502 21:29:23.007849 11554 rebalancer_tool.cc:1074] tablet 4674e376bf7440cda619784491685b44: '525a093ea5bf40dc9bbf9dc39895d670' -> '6ef0beaf2f6941afb6ea9f9f274517fe' move scheduled
I20240502 21:29:23.279397 11554 rebalancer_tool.cc:1084] tablet e46065d94e874293a836bc74241b8e06: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> 'd10e9e156ba74a97a221c85b68ab8bc4' move ignored: Illegal state: tablet e46065d94e874293a836bc74241b8e06: replica d10e9e156ba74a97a221c85b68ab8bc4 is already present
I20240502 21:29:23.279641 11554 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 21:29:24.499892 13950 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.27.196.126:34625, user_credentials={real_user=slave}} blocked reactor thread for 46245.7us
W20240502 21:29:25.554589 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c0a0ef39a2a246449eb582cc1d993484: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:25.554778 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef412fa8fe9249729cb99879a5c8343a: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:25.559707 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e8eef05f850243d0bdc7e1899f8c6139: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:25.559924 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a237d7bda970485480e98020c6ac738f: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:25.560042 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5802da0404e74a07bca290cdc59b279b: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:25.560184 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f52c6c3804274777abfa00bb33ae9b16: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:25.560410 11554 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 445d2e6ec7154b80abec5fff346b8f5e: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server 6ef0beaf2f6941afb6ea9f9f274517fe
I20240502 21:29:25.655982 11554 rebalancer_tool.cc:1074] tablet a7c6b881c9ef4cdca787f065778e526f: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '' move scheduled
I20240502 21:29:25.730135 11554 rebalancer_tool.cc:1074] tablet 58c7004ceb044f5fa9dec54910046b69: '6ef0beaf2f6941afb6ea9f9f274517fe' -> '' move scheduled
I20240502 21:29:25.955307 11554 rebalancer_tool.cc:1074] tablet f0259f01acb249dd99c662151dc21916: '5543ddab5b484642b139b7c848932a58' -> '' move scheduled
W20240502 21:29:31.377310 11632 kernel_stack_watchdog.cc:198] Thread 11554 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 259ms:
Kernel stack:
(could not read kernel stack)
User stack:
    @     0x7fa33b7c1980  (unknown) at ??:0
    @           0x90ef77  __sanitizer::internal_sched_yield() at ??:?
    @           0x87462d  __interceptor_pthread_create at ??:?
    @     0x7fa33842fd11  kudu::Thread::StartThread() at ??:0
    @           0xa4d187  kudu::Thread::Create() at ??:?
    @     0x7fa33f087530  kudu::rpc::ReactorThread::Init() at ??:0
    @     0x7fa33f092797  kudu::rpc::Reactor::Init() at ??:0
    @     0x7fa33f03b5c5  kudu::rpc::Messenger::Init() at ??:0
    @     0x7fa33f03a9d1  kudu::rpc::MessengerBuilder::Build() at ??:0
    @     0x7fa346678db5  kudu::client::KuduClientBuilder::Build() at ??:0
    @     0x7fa34c31ed5c  kudu::tools::CreateKuduClient() at ??:0
    @     0x7fa34d25391b  kudu::tools::DoChangeConfig() at ??:0
    @     0x7fa34d2518d7  kudu::tools::CheckCompleteMove() at ??:0
    @     0x7fa34d1d61df  kudu::tools::RebalancerTool::AlgoBasedRunner::UpdateMovesInProgressStatus() at ??:0
    @     0x7fa34d1cd82b  kudu::tools::RebalancerTool::RunWith() at ??:0
    @     0x7fa34d1ca14c  kudu::tools::RebalancerTool::Run() at ??:0
W20240502 21:29:31.445344 11554 thread.cc:636] rpc reactor (reactor) Time spent creating pthread: real 0.637s	user 0.058s	sys 0.130s
W20240502 21:29:31.445744 11554 thread.cc:602] rpc reactor (reactor) Time spent starting thread: real 0.637s	user 0.058s	sys 0.130s
I20240502 21:29:34.228063 11554 rebalancer_tool.cc:1123] tablet 58c7004ceb044f5fa9dec54910046b69: '6ef0beaf2f6941afb6ea9f9f274517fe' -> '' move is abandoned: Not found: Server with UUID 6ef0beaf2f6941afb6ea9f9f274517fe not a member of the config. RaftConfig: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:29:34.433872 11554 rebalancer_tool.cc:1136] tablet 51936fb1f678458a8795f8246590addb: '5543ddab5b484642b139b7c848932a58' -> '' move completed: OK
I20240502 21:29:42.533684 11554 rebalancer_tool.cc:1136] tablet 07d693b680a54e12871121737e3da744: '525a093ea5bf40dc9bbf9dc39895d670' -> '6ef0beaf2f6941afb6ea9f9f274517fe' move completed: OK
I20240502 21:29:48.282455 11554 rebalancer_tool.cc:1136] tablet 79dedaa327b6479e8325f91481c1688e: '19f162e164854f2da623a4bb8f9e6bf6' -> 'e0de0545fa0547d3929111e990e6d1b2' move completed: OK
I20240502 21:29:51.554951 11554 rebalancer_tool.cc:1136] tablet 41ee481a3ece47a0905e93c66dbf949d: '5543ddab5b484642b139b7c848932a58' -> '4cbb5b06a6a44eb6b2145cd538b01af2' move completed: OK
I20240502 21:29:51.909400 11554 rebalancer_tool.cc:1136] tablet aeee59c5512d448d8f986f959dce1cc8: '19f162e164854f2da623a4bb8f9e6bf6' -> 'd10e9e156ba74a97a221c85b68ab8bc4' move completed: OK
I20240502 21:29:52.013919 11554 rebalancer_tool.cc:1136] tablet 1497b781f36c43b39d22a05bb524c59d: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> 'e0de0545fa0547d3929111e990e6d1b2' move completed: OK
I20240502 21:29:54.783805 11554 rebalancer_tool.cc:1074] tablet cdf72d25f0f847e5998acbc53979e64d: '19f162e164854f2da623a4bb8f9e6bf6' -> '' move scheduled
I20240502 21:29:55.098533 11554 rebalancer_tool.cc:1136] tablet cdf72d25f0f847e5998acbc53979e64d: '19f162e164854f2da623a4bb8f9e6bf6' -> '' move completed: OK
I20240502 21:29:55.223795 11554 rebalancer_tool.cc:1136] tablet f0259f01acb249dd99c662151dc21916: '5543ddab5b484642b139b7c848932a58' -> '' move completed: OK
I20240502 21:29:55.441762 11554 rebalancer_tool.cc:1136] tablet 4674e376bf7440cda619784491685b44: '525a093ea5bf40dc9bbf9dc39895d670' -> '6ef0beaf2f6941afb6ea9f9f274517fe' move completed: OK
I20240502 21:29:55.595600 11554 rebalancer_tool.cc:1136] tablet 4d1a31513bb64578a85adf1f622e5bc0: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '' move completed: OK
I20240502 21:29:55.700323 11554 rebalancer_tool.cc:1136] tablet a2f3edaf1c874cefb820f027f90d1185: '5543ddab5b484642b139b7c848932a58' -> '4cbb5b06a6a44eb6b2145cd538b01af2' move completed: OK
I20240502 21:29:55.847783 11554 rebalancer_tool.cc:1136] tablet a7c6b881c9ef4cdca787f065778e526f: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '' move completed: OK
I20240502 21:29:55.923099 11554 rebalancer_tool.cc:1136] tablet 32c306d724954a16be21a34aa4cc0b5f: '525a093ea5bf40dc9bbf9dc39895d670' -> '6ef0beaf2f6941afb6ea9f9f274517fe' move completed: OK
I20240502 21:29:56.043507 11554 rebalancer_tool.cc:1136] tablet 8a2e06253c51405fa9b6bc9e87844760: '19f162e164854f2da623a4bb8f9e6bf6' -> 'd10e9e156ba74a97a221c85b68ab8bc4' move completed: OK
I20240502 21:29:56.139101 11554 rebalancer_tool.cc:1074] tablet be34ec89876d42b69efe1878c0d9bc19: '5543ddab5b484642b139b7c848932a58' -> '' move scheduled
I20240502 21:29:56.214038 11554 rebalancer_tool.cc:1074] tablet de3287296c4547688388388dd4c7e638: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '' move scheduled
I20240502 21:29:56.339948 11554 rebalancer_tool.cc:1074] tablet 3d1a5050d3f944969a560994a23700ad: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move scheduled
I20240502 21:29:56.513465 11554 rebalancer_tool.cc:1074] tablet 3caa36212ba444aa88d153ff9f90122e: '19f162e164854f2da623a4bb8f9e6bf6' -> '' move scheduled
I20240502 21:29:57.251376 11554 rebalancer_tool.cc:1136] tablet 3caa36212ba444aa88d153ff9f90122e: '19f162e164854f2da623a4bb8f9e6bf6' -> '' move completed: OK
I20240502 21:29:57.585975 11554 rebalancer_tool.cc:1136] tablet 3d1a5050d3f944969a560994a23700ad: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move completed: OK
I20240502 21:29:57.926736 11554 rebalancer_tool.cc:1136] tablet de3287296c4547688388388dd4c7e638: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '' move completed: OK
I20240502 21:30:00.113062 11554 rebalancer_tool.cc:1074] tablet 8129d13163f64368b731ba4eaeaaa81c: '19f162e164854f2da623a4bb8f9e6bf6' -> '' move scheduled
I20240502 21:30:00.256199 11554 rebalancer_tool.cc:1074] tablet 87f35aff2b104c01a9d9d48a7e9c9b23: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move scheduled
I20240502 21:30:00.372195 11554 rebalancer_tool.cc:1074] tablet f82c35077224414cb284472d49d7de4f: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '' move scheduled
I20240502 21:30:09.213039 11554 rebalancer_tool.cc:1136] tablet 8129d13163f64368b731ba4eaeaaa81c: '19f162e164854f2da623a4bb8f9e6bf6' -> '' move completed: OK
I20240502 21:30:09.394202 11554 rebalancer_tool.cc:1136] tablet be34ec89876d42b69efe1878c0d9bc19: '5543ddab5b484642b139b7c848932a58' -> '' move completed: OK
I20240502 21:30:09.502931 11554 rebalancer_tool.cc:1074] tablet cffcb4f7f7754262beeb06168097f6dc: '5543ddab5b484642b139b7c848932a58' -> '' move scheduled
I20240502 21:30:09.708202 11554 rebalancer_tool.cc:1074] tablet f9ae8fac693d4d68b6741f2509e1265e: '5543ddab5b484642b139b7c848932a58' -> '' move scheduled
I20240502 21:30:09.817711 11554 rebalancer_tool.cc:1136] tablet f9ae8fac693d4d68b6741f2509e1265e: '5543ddab5b484642b139b7c848932a58' -> '' move completed: OK
I20240502 21:30:09.954106 11554 rebalancer_tool.cc:1136] tablet cffcb4f7f7754262beeb06168097f6dc: '5543ddab5b484642b139b7c848932a58' -> '' move completed: OK
I20240502 21:30:10.021117 11554 rebalancer_tool.cc:1123] tablet f82c35077224414cb284472d49d7de4f: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:10.125454 11554 rebalancer_tool.cc:1136] tablet 87f35aff2b104c01a9d9d48a7e9c9b23: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move completed: OK
Illegal state: tablet server 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): unacceptable health status UNAVAILABLE

I20240502 21:30:12.048527  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:12.096035  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:12.116914  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:12.122591 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:12.127072 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:30:12.134639 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:12.136520  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:12.189524 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:12.207916  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:12.231396  9173 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 60026.8us
W20240502 21:30:12.237756  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 21:30:12.262459 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:12.290953  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:12.330060  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:12.393383 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:12.434094 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:12.444075  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:12.721139  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:12.722021  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:30:12.737771 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:12.757130 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:12.766878  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:30:12.770463  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 21:30:12.774329  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:12.789791  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:12.804600  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:12.815096 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:12.822911 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:12.843899  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:30:12.869273  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:30:12.893811  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:30:12.949298  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 21:30:13.002099 11093 consensus_queue.cc:237] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:30:13.004252  9507 raft_consensus.cc:1261] T de3287296c4547688388388dd4c7e638 P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Refusing update from remote peer d10e9e156ba74a97a221c85b68ab8bc4: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:30:13.004876 20522 consensus_queue.cc:1035] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:30:13.008121 20716 raft_consensus.cc:2932] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:30:13.008816  9507 raft_consensus.cc:2932] T de3287296c4547688388388dd4c7e638 P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:30:13.013589  8958 catalog_manager.cc:5484] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4 reported cstate change: config changed from index 5 to 7, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) evicted. New cstate: current_term: 2 leader_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:30:13.042541  9085 tablet_service.cc:1514] Processing DeleteTablet for tablet de3287296c4547688388388dd4c7e638 with delete_type TABLET_DATA_TOMBSTONED (TS 5543ddab5b484642b139b7c848932a58 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47814
I20240502 21:30:13.079324 20729 tablet_replica.cc:330] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58: stopping tablet replica
I20240502 21:30:13.081120 20729 raft_consensus.cc:2229] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:30:13.081398 20729 raft_consensus.cc:2258] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:30:13.087858 20729 ts_tablet_manager.cc:1875] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:30:13.095743 20729 ts_tablet_manager.cc:1888] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 21:30:13.098920 11093 consensus_queue.cc:237] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:30:13.103242 20729 log.cc:1196] T de3287296c4547688388388dd4c7e638 P 5543ddab5b484642b139b7c848932a58: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/wals/de3287296c4547688388388dd4c7e638
I20240502 21:30:13.104873  8945 catalog_manager.cc:4830] TS 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): tablet de3287296c4547688388388dd4c7e638 (table rebalance_test_table_0 [id=445d2e6ec7154b80abec5fff346b8f5e]) successfully deleted
W20240502 21:30:13.106967 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 21:30:13.117482  9507 raft_consensus.cc:1261] T de3287296c4547688388388dd4c7e638 P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Refusing update from remote peer d10e9e156ba74a97a221c85b68ab8bc4: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 21:30:13.120604 20716 consensus_queue.cc:1035] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:30:13.126531 20522 raft_consensus.cc:2932] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:30:13.127633  9507 raft_consensus.cc:2932] T de3287296c4547688388388dd4c7e638 P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:30:13.133025  8946 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet de3287296c4547688388388dd4c7e638 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20240502 21:30:13.139847 11025 consensus_peers.cc:485] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4 -> Peer 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857): Couldn't send request to peer 4cbb5b06a6a44eb6b2145cd538b01af2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de3287296c4547688388388dd4c7e638. This is attempt 1: this message will repeat every 5th retry.
W20240502 21:30:13.140951 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:13.138892  8963 catalog_manager.cc:5484] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4 reported cstate change: config changed from index 7 to 8, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New cstate: current_term: 2 leader_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
W20240502 21:30:13.219738  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 21:30:13.254843  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:13.279997  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:13.282588 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:13.318279  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:13.341526 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:13.364260 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:13.379285  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:13.546094  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:13.555155 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:13.555442 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:13.556962  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:13.559376 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:13.565330  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:13.590267  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:13.651070 20789 ts_tablet_manager.cc:902] T de3287296c4547688388388dd4c7e638 P 4cbb5b06a6a44eb6b2145cd538b01af2: Initiating tablet copy from peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:30:13.667409 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:13.740160 20789 tablet_copy_client.cc:308] T de3287296c4547688388388dd4c7e638 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.69:42049
I20240502 21:30:13.762936 11114 tablet_copy_service.cc:140] P d10e9e156ba74a97a221c85b68ab8bc4: Received BeginTabletCopySession request for tablet de3287296c4547688388388dd4c7e638 from peer 4cbb5b06a6a44eb6b2145cd538b01af2 ({username='slave'} at 127.27.196.71:46913)
I20240502 21:30:13.763285 11114 tablet_copy_service.cc:161] P d10e9e156ba74a97a221c85b68ab8bc4: Beginning new tablet copy session on tablet de3287296c4547688388388dd4c7e638 from peer 4cbb5b06a6a44eb6b2145cd538b01af2 at {username='slave'} at 127.27.196.71:46913: session id = 4cbb5b06a6a44eb6b2145cd538b01af2-de3287296c4547688388388dd4c7e638
I20240502 21:30:13.769502 11114 tablet_copy_source_session.cc:205] T de3287296c4547688388388dd4c7e638 P d10e9e156ba74a97a221c85b68ab8bc4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:30:13.772058 20789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de3287296c4547688388388dd4c7e638. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:30:13.779496 20789 tablet_copy_client.cc:783] T de3287296c4547688388388dd4c7e638 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Starting download of 0 data blocks...
I20240502 21:30:13.779994 20789 tablet_copy_client.cc:647] T de3287296c4547688388388dd4c7e638 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:30:13.783066 20789 tablet_copy_client.cc:523] T de3287296c4547688388388dd4c7e638 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:30:13.788705 20789 tablet_bootstrap.cc:492] T de3287296c4547688388388dd4c7e638 P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap starting.
I20240502 21:30:13.875558 20789 tablet_bootstrap.cc:492] T de3287296c4547688388388dd4c7e638 P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:30:13.876121 20789 tablet_bootstrap.cc:492] T de3287296c4547688388388dd4c7e638 P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap complete.
I20240502 21:30:13.876482 20789 ts_tablet_manager.cc:1367] T de3287296c4547688388388dd4c7e638 P 4cbb5b06a6a44eb6b2145cd538b01af2: Time spent bootstrapping tablet: real 0.088s	user 0.075s	sys 0.003s
I20240502 21:30:13.877306 20789 raft_consensus.cc:348] T de3287296c4547688388388dd4c7e638 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:30:13.877552 20789 raft_consensus.cc:729] T de3287296c4547688388388dd4c7e638 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbb5b06a6a44eb6b2145cd538b01af2, State: Initialized, Role: FOLLOWER
I20240502 21:30:13.877831 20789 consensus_queue.cc:260] T de3287296c4547688388388dd4c7e638 P 4cbb5b06a6a44eb6b2145cd538b01af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:30:13.878417 20789 ts_tablet_manager.cc:1398] T de3287296c4547688388388dd4c7e638 P 4cbb5b06a6a44eb6b2145cd538b01af2: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 21:30:13.882489 11114 tablet_copy_service.cc:342] P d10e9e156ba74a97a221c85b68ab8bc4: Request end of tablet copy session 4cbb5b06a6a44eb6b2145cd538b01af2-de3287296c4547688388388dd4c7e638 received from {username='slave'} at 127.27.196.71:46913
I20240502 21:30:13.882689 11114 tablet_copy_service.cc:433] P d10e9e156ba74a97a221c85b68ab8bc4: ending tablet copy session 4cbb5b06a6a44eb6b2145cd538b01af2-de3287296c4547688388388dd4c7e638 on tablet de3287296c4547688388388dd4c7e638 with peer 4cbb5b06a6a44eb6b2145cd538b01af2
I20240502 21:30:13.938414 20474 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:30:13.938649 20474 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:30:13.939498 20474 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:30:13.941864 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:13.942219 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
I20240502 21:30:13.942345 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:30:13.942746 20474 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 21:30:14.040163 11363 raft_consensus.cc:1203] T de3287296c4547688388388dd4c7e638 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8]   Dedup: 2.8->[]
I20240502 21:30:14.073359  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:14.092088 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:14.111702  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:14.116076  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:30:14.120304  8946 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240502 21:30:14.131126 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:14.164366 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:14.187645  8946 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 67353.4us
W20240502 21:30:14.189615  8946 catalog_manager.cc:4573] TS 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4d1a31513bb64578a85adf1f622e5bc0: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:14.233160  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:14.243880 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:14.317381  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 21:30:14.441190  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:14.465878  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:14.485141 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:14.494810  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:14.495087 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:14.507215  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:14.509162 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:14.600183 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:14.612454  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 21:30:14.829133  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:14.894161 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:14.911207  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:14.914362 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:14.917833 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:14.928428  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:14.947170  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:30:14.981102 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:15.001490 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:15.005165  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:15.035473  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 21:30:15.091554  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:15.122875  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:30:15.159193  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:30:15.220903  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 21:30:15.236294  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:15.240597 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:15.248319 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:15.426769  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:30:15.448021  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:30:15.449009  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:30:15.510026  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 21:30:15.553931 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:15.572418  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:15.602643  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:15.613711  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 21:30:15.618568 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:15.619823 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:15.619897  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 21:30:15.664577  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:15.667433 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:15.675818 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 22 similar messages]
I20240502 21:30:15.679461  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:15.680789  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:15.687083  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:30:15.688992 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:30:15.886850  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 21:30:15.928170  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:15.933136  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:15.944695  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:15.948917 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:15.967850 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:15.973954  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:15.989048 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:15.995252 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:16.269124  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:16.296797 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:16.303186 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:16.304504  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:16.306063 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:16.309577  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:16.313268  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:16.403633 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:16.581159 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:16.592130  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:16.605769  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:16.610220 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:16.618963  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:16.633225  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:16.651868 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:16.680560  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:30:16.917956 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:16.963851  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:16.974797  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:16.975442 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:16.982836  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:16.996451  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:17.012471 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:17.057551 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:17.244688  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 21:30:17.339018  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:17.339587 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:17.340580  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:17.361152  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:17.367679 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:17.371595 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:17.386322  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 21:30:17.387598  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:17.407243 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:17.575093  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:30:17.611596  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 21:30:17.654994 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:17.659629  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:17.696172  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:17.701037  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:17.705710  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:17.708721 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:17.729135 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:17.753572  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:30:17.912252  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 21:30:17.969664  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:30:17.976603 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:17.978252  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:17.980279 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:30:17.988567  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 21:30:17.991148 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:17.994275 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:17.995893 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 21:30:18.000216  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:18.023816  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:30:18.085616  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:30:18.200414  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 21:30:18.292735  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:18.297511  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:18.299701 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:18.307416 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:18.349056 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:18.362515  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:18.374596  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:30:18.485162 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:18.558027 20474 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:30:18.558276 20474 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:30:18.559036 20474 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:30:18.560768 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:18.562114 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
I20240502 21:30:18.562258 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:30:18.562925 20474 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 21:30:18.624953  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:18.630458  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:18.635157  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:18.635869 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:18.668076 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:18.669019 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:18.703640  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:18.939307  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:18.952870  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:18.955427  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:18.959887 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:18.974761 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:18.993921 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:19.009174  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:19.040419 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:19.172426  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 21:30:19.264727  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:19.273800 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:19.279961  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:19.290786 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:19.306754  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:19.325798  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:19.349812 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:19.415331 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:19.504916  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 21:30:19.537092  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:19.564744  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:19.571046 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:19.574329  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:19.582144 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:19.588729  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:19.596230 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:19.751005  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:30:19.792408 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:19.849128  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:19.849915  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:19.878890 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:19.883222 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:19.895030  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:19.922067  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:19.948068 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:19.957307  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:30:20.115162  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 21:30:20.230589  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:20.232440  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 21:30:20.245420  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:20.246762 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:20.250335 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:20.268566  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:20.272292 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:20.294329  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:30:20.307965  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:30:20.379567 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:20.505546  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:30:20.507889  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:30:20.520982  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 21:30:20.546283  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:20.548300 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:20.563908  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:20.570811  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:20.584926 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:20.585582 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:20.625093  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 21:30:20.636631  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:30:20.637908 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 21:30:20.643260  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:20.723757  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:20.724995  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 21:30:20.868201  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:20.872538 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 21:30:20.876375 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:20.904525 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:20.922716  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:20.926283  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:20.926879 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:20.936502 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:20.952483  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:21.212960 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:21.213533  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:21.215781  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:21.248664  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:21.265030 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:21.269049 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:21.293918  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:21.340101  9173 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 76610.5us
W20240502 21:30:21.493587 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:21.508721  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:21.516290 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:21.520630  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:21.523036 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:21.536522  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:21.543081 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:21.560395  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:21.822758  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 21:30:21.823976  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:21.838088  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:21.839465 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:21.848124  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:21.880164 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:21.881143 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:21.904739  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:21.927037  9040 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.67:43203, user_credentials={real_user=slave}} blocked reactor thread for 44951.6us
W20240502 21:30:21.990763 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:22.029304  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:30:22.065660  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 21:30:22.145058  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:22.145447  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:22.169687  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:22.200997 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:22.222951 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:22.276154 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:22.286646  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:22.385967 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:22.396080  8946 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:22.397668  8946 catalog_manager.cc:4573] TS 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4d1a31513bb64578a85adf1f622e5bc0: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:22.508893  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:22.523825 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:22.528952 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:22.538195  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:30:22.542481  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 21:30:22.554227 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:22.566754  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:22.568843  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:22.722719  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:30:22.746841  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 21:30:22.859091  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:22.861128 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:22.864813  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:22.864881 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:22.876399  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:22.882279  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:22.900099 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:22.938892  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:30:22.992908  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:30:22.995016 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:23.001160  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:30:23.121325 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 21:30:23.156761  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:23.177610 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:23.199751  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:30:23.218962  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 21:30:23.242848  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:23.252518 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:23.275404  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:23.280159 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:23.483491 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:23.513072  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:23.517982 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:23.539660  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:23.540380 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:23.561834  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:23.573048 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:23.573710  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:23.625684 20474 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:30:23.625886 20474 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:30:23.626693 20474 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:30:23.628913 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:23.629169 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
I20240502 21:30:23.629251 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:30:23.629468 20474 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 21:30:23.841615  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:23.847607 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:23.856454  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:23.856765  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:23.866076 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:23.883670 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:23.915546  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:24.046234 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:24.182766  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:24.196110 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:24.220619  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:24.227511  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:24.273193  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:24.279269 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:24.279479 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:24.366720  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:30:24.426662  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 21:30:24.448256  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:24.461496 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:24.463209  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:24.463515  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:24.465171 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:24.487061 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:24.541503 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:24.556339  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:24.624619  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 21:30:24.776553  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:24.787431  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:24.810664 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:24.816922 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:24.821853 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:24.836104  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:24.847568  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:30:24.900061  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:30:25.101224 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:25.127185  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:25.152952  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:25.160446  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:25.169519 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:25.170794 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:25.178577  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:25.189656 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:25.224982  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:30:25.352468  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:30:25.428680  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 21:30:25.490664  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:25.491537 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:25.503152  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:25.503971 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:25.535487 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:30:25.559551  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 21:30:25.564867 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:25.567875  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:25.568850 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:25.578459  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:30:25.579493  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:30:25.598220  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:30:25.625406  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 21:30:25.632967  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:25.757079  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 21:30:25.810899  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:25.824296  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:25.837282 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:25.858496  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:25.858767 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:25.908551  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:25.914690 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:26.026744  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:30:26.070891 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:30:26.072621 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:26.125087  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:26.133447 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:26.156406  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:26.173841 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:26.230875  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:26.270735 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:26.275794  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:26.508396  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:26.532060 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:26.558468  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:26.562582 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:26.605175 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:26.608726  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:30:26.612687 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:26.647053  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:26.700425  9173 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 135754us
W20240502 21:30:26.713187  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:30:26.810040  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 21:30:26.944329 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:26.945549  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:26.977589 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:26.986037  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:27.004547 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:27.005909  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:27.048517  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:27.098956  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:30:27.113528 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:27.339293  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:27.343856  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:27.367558 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:27.377131 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:27.378486 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:27.379766  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:27.410079  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:30:27.587191  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:30:27.625630  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:30:27.661147 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:27.731920  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:27.745366  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:27.751838 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:27.759135  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:30:27.783496  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 21:30:27.796526 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:27.819766  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:27.829828 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:27.871071 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:30:27.927862  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:30:27.928907  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 21:30:28.058845  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:28.061264  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 21:30:28.070982 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:28.074376  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:28.075382  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:28.096907  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:28.104007 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:28.117643 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:28.190215 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:28.212368  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 21:30:28.419171  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:28.432497  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:28.443174  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:28.446470 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:28.455519 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:28.460135 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:28.467442  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:28.755081  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:28.788017 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:28.812750  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:28.816078  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:28.816468 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:28.841558 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:28.846626 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:28.857568  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:28.952282  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 21:30:29.087544  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:29.113343  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:29.126200 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:29.159081  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:29.173266 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:29.174393 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:29.205349  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:29.267856  9040 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.67:43203, user_credentials={real_user=slave}} blocked reactor thread for 55660.3us
W20240502 21:30:29.345132 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:29.411406  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 21:30:29.492890  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:29.503651  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:29.506841 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:29.507428 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:29.517230  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:29.524010 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:29.569254  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:29.827029 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:29.862432  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:29.890125  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:29.890417 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:29.891851  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:29.897125  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 21:30:29.913134 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:29.927974 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:29.950276  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:30:30.020836  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:30:30.110994  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 21:30:30.156854  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:30.164839 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:30.206018  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:30.225507  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:30.253103 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:30.268618 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:30.273773  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:30.367165 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:30:30.429258 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:30.430298  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:30:30.485532  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:30:30.506397  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 21:30:30.544764  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:30.549770 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:30.558539  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:30.573590  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:30.612010 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:30.656822 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:30.659921  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:30.661303  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240502 21:30:30.784657  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 21:30:30.883306  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:30.903481  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:30.909024 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:30.910087  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:30.936734 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:30.938956 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:30.941577  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:30.963652 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:30.986310  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:30:31.012395  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 21:30:31.031509  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 21:30:31.276103  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:31.281301  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:31.285039  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:31.288573 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:31.306003 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:31.328130  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 21:30:31.342572 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:31.374192 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:30:31.376008 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:31.401708  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:31.496634  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 21:30:31.616609  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:31.635094  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:31.638103 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:31.642879  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:31.645658 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:31.669598  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:31.675009 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:31.758188  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:30:31.995386 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:32.242072  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:32.254262  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:32.254370 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:32.280261  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:32.307745 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:32.314776 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:32.316505  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:32.359158  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:30:32.372239  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:30:32.468068 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:32.597638  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 21:30:32.737435  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:32.737674 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 21:30:32.740139 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:32.781772  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:32.787436  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:32.815688 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:32.828330  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 21:30:32.895850 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:32.939962  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 21:30:32.955176  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:32.988437 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:33.013654  9445 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.69:42049, user_credentials={real_user=slave}} blocked reactor thread for 55430.8us
I20240502 21:30:33.107121  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:33.116779  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 21:30:33.128517 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:33.144858 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:33.159186  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:33.167522 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:33.171535  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:33.194080  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:33.366115  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:30:33.382593 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:33.460441  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:33.474225 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:33.475008 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:33.476612  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:33.479782  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:33.493541  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:33.494244 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:33.629649  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 21:30:33.780045  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:30:33.794143 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:33.799760  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:33.802479  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:33.805255 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:33.807615 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:33.808691 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:33.877187  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:34.028527  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 21:30:34.126777  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:34.153713 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:34.154188  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:34.166865 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:34.168943 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:34.172044  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:34.188669  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:30:34.206362 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:34.222164  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:30:34.292629  9173 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 69815us
I20240502 21:30:34.498036  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:34.500041 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:34.502514  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:34.507848 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:34.519372 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:34.521539  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:34.562216  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:30:34.657965 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:34.733956  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 21:30:34.811637  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:34.821002  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:34.840623 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:34.849424  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:34.850490 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:34.871299  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:34.878227 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:34.879186  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:30:34.922508  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:30:35.085292 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:35.148224  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:35.168581  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:30:35.174273 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 21:30:35.183557 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:35.199628  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:35.239953 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:35.266973 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:35.301959  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:35.511637  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:35.517592  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:35.518033 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:35.547386  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:35.550272 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:35.553946 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:35.563443  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:30:35.567767  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 21:30:35.572994  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:35.597862 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:35.692260  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:30:35.759996  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 21:30:35.803589  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 21:30:35.864379  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:35.869371  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:35.878031 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:35.893055  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:35.896828 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:35.911711 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:35.946152  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:35.976150  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:30:36.052990  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 21:30:36.078393  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 21:30:36.185946  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:36.192623 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:36.200115  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:36.209089 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:36.232174  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:36.248477 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:36.277052 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:36.303455  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:36.485574  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:36.502605 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:30:36.514596 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 21:30:36.545938  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:36.548045 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:36.552098  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:36.570895 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:36.589206  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:36.597121  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:36.623376 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:36.634042  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:30:36.663376  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 21:30:36.824136 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:36.829197  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:36.842861  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:36.852605  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:36.854158 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:36.854808  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:36.863667 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:36.958262  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:30:37.154510  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 21:30:37.156309 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:37.172700  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:37.172776  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:37.173739  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:37.212805  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:37.219127 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:37.224345  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:30:37.235670 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:37.259995 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:37.293756  9173 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 84603.1us
I20240502 21:30:37.478365 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:37.479593  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:37.490829  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:37.511569 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:37.512270 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:37.525801  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:37.532130  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:30:37.532665 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:30:37.664819 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:37.789814  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:37.797720 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:37.799361  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:37.803256  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:37.815863 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:37.817941  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:37.830821 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:37.957224  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 21:30:38.076345  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:30:38.102226  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 21:30:38.102370 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:38.119930 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:38.122676  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:38.124069  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:38.142078 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:30:38.204453 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:38.206569  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:38.211987  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:30:38.317492  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 21:30:38.418408  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:38.421440 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:38.424787  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:38.440685  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:38.466428  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:38.466838 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:38.505836 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:38.521134  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:30:38.661840 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:38.773903  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:38.774471  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:38.774746 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:38.812678  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:38.834108 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:38.864850 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:38.895380  8946 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:38.903839  8946 catalog_manager.cc:4573] TS 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4d1a31513bb64578a85adf1f622e5bc0: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:38.941882  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:39.072626  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:30:39.192445  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 21:30:39.205914  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:39.206470  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:39.213727 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:39.231719 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:39.246966  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:39.248147 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:39.294261 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:39.317026  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:39.417201  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 21:30:39.559019  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:39.562261 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:39.564635  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:39.567936  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:39.594975 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:39.596226  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:39.609077 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:39.612020 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:39.617660  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 21:30:39.654213 23888 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:30:39.654459 23888 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:30:39.655436 23888 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:30:39.658411 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
W20240502 21:30:39.658639 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:39.658744 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:30:39.659067 23888 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 21:30:39.867079  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:39.873186 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:39.874102  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:39.889544  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:39.914672 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:39.928779 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:39.931373  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:39.988041  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:30:40.001529 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 21:30:40.185254  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:40.207705 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:40.222855 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:40.257617  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:40.273154 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:40.276535  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:40.318049  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:40.354581 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:40.485891  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 21:30:40.570838  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:40.572824 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:30:40.601116  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 21:30:40.603864  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:40.610960  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:30:40.615445  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 21:30:40.620327  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:40.620777 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:40.621174 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:40.699918 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:40.712566  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:30:40.837432  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 21:30:40.881147  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:40.895143  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:40.913834 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:40.915998  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:40.918720 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:40.935868  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 21:30:40.944882 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:40.978363  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:41.169770  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:41.185191 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:41.210194  8951 maintenance_manager.cc:419] P 7eba6d8fd1d94cf8b98795fe2505a077: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033402
W20240502 21:30:41.213138 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:41.228467  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:30:41.250728  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 21:30:41.268160  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:41.282095 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:41.308332 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:41.318501  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:41.362730  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 21:30:41.392292  8950 maintenance_manager.cc:643] P 7eba6d8fd1d94cf8b98795fe2505a077: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.181s	user 0.042s	sys 0.008s Metrics: {"bytes_written":20162,"cfile_init":1,"dirs.queue_time_us":8001,"dirs.run_cpu_time_us":419,"dirs.run_wall_time_us":4956,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1824,"lbm_writes_lt_1ms":28,"peak_mem_usage":0,"rows_written":43,"thread_start_us":8640,"threads_started":2,"wal-append.queue_time_us":386}
I20240502 21:30:41.394503  8951 maintenance_manager.cc:419] P 7eba6d8fd1d94cf8b98795fe2505a077: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7099 bytes on disk
I20240502 21:30:41.400318  8950 maintenance_manager.cc:643] P 7eba6d8fd1d94cf8b98795fe2505a077: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4}
W20240502 21:30:41.525671  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 21:30:41.561766  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:41.565040 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:41.601289  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:41.602363 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:41.604481 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:41.630895  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:41.640800  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:30:41.676048 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:30:41.728094  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:41.818337 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:41.858822  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 21:30:41.881268  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:41.886200  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:41.903707  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:41.906563 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:41.910780 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:41.919592 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:41.924010  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:41.992022  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:30:42.041617  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:30:42.215497 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:42.216569  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:42.234346 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:42.257395  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:42.258184 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:42.296419  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:42.345531 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:42.371384  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:42.593464  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:42.597224  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:42.598001 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:42.615469  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:42.633931 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:42.669910 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:42.673663  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:42.702256 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:30:42.704319 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:42.828311  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:30:42.946478  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 21:30:42.967558  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:42.989972 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:42.991873 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:42.995936  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:43.000231 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:43.001231  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:43.002557  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:43.137893  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:30:43.202106  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:30:43.216603  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:30:43.217978  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 21:30:43.293846 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:30:43.299244 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:43.303475  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:43.304723  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:43.331400  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:43.347787 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:43.357789 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:43.377193  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:43.622215  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:43.623296  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:43.623644 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:43.643016 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:43.646534  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:43.677224  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:43.688930 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:43.842595 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:43.842558  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 21:30:43.992969 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:43.995082  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:43.998502  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:44.001621  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:44.004472 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:44.015944  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:44.026211 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:44.140138  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 21:30:44.308151  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:44.337539  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:30:44.345461  9039 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.69:42049, user_credentials={real_user=slave}} blocked reactor thread for 46467.3us
I20240502 21:30:44.355931 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:44.356995 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:44.360085  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:44.366039 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:30:44.368610 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:44.372207  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:30:44.380698  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:30:44.470600  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 21:30:44.640620  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:44.643414 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:44.646395  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:44.662078  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:44.662182 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:44.667654 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:44.688815  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:30:44.936672 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:44.952531  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:44.954054 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:44.987567 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:45.000182  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:45.001425  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:45.035683  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:45.037312 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:45.158252  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:30:45.284443  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:30:45.287521 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 21:30:45.289196 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:45.291438  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:45.312975  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:45.322685 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:45.325662  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:45.327450 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:45.335088  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:45.339643 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:45.596865  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:45.602260  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:45.604036  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:45.609537 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:45.611742 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:45.623759 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:45.636215  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:45.746625  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:30:45.775463  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:30:45.794723  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:30:45.800128  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:30:45.815172 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:45.930953 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:45.937284 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:45.938612  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:45.947089 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:45.950783  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:45.950786  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:45.971455  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:45.984156  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 21:30:46.206544 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:46.245064  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:46.265165  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:46.265861 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:46.275415 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:46.276803  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:46.286978 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:46.331880  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:46.425498  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:30:46.429240  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 21:30:46.525295 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:46.526932  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:46.544584  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:46.551956  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:46.576926 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:46.606072  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:46.612469  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 21:30:46.636265 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:46.714763  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:30:46.784050 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:30:46.785238 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:46.871479 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:46.873601  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:46.878182  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:46.880607  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:46.904417 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:46.915829 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:46.945858  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:46.976356  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:30:47.002673  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:47.023051  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:30:47.172017 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:47.213864  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:47.213958 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:47.227011  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:47.256064  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:47.295636 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:47.297204  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:47.324981 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:47.509581  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 21:30:47.548676  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:47.557564  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:47.568419 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:47.573662 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:47.585778 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:47.586917  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:47.609031 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:47.631176  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:47.837879  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:47.856165  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:47.859400 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:47.867609  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:47.882747 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:47.896288 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:47.913857 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 21:30:47.928645  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:47.980007  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:30:48.123016  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:30:48.128008 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:48.167622 25065 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:30:48.167891 25065 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:30:48.168748 25065 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:30:48.170547 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
W20240502 21:30:48.172138 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:48.172287 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:30:48.173135 25065 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 21:30:48.232180  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 21:30:48.335333  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:48.341197  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:48.348296  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:48.348367 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:48.358938 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:48.389935  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:30:48.413777  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 21:30:48.452075 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:48.469532  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:48.648406 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:48.738746  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:48.758814  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:48.764642 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:48.768218 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:48.774605  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:48.809276  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:48.832547 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:48.839298  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:30:48.983354  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 21:30:49.042663  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:49.064087  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:30:49.079298 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:49.079367  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:49.109232 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:49.125802  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:49.130285 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:49.157698 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:30:49.309340  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 21:30:49.383234 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:49.393110  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:49.405429  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 21:30:49.418360 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:49.421001 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:49.423156  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:49.429287  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:49.484447  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:30:49.500006  9174 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.67:43203, user_credentials={real_user=slave}} blocked reactor thread for 55698.8us
W20240502 21:30:49.541249  9172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.68:40931, user_credentials={real_user=slave}} blocked reactor thread for 102624us
W20240502 21:30:49.552559 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:49.748840  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:49.750504 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:49.758984  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:49.764408 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:49.766381 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:49.770102  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:49.865036  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:49.981835 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:50.044790  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:50.050158 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:50.063622 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:50.065613 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:50.069569  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:50.095309  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:50.100500  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:50.258601  9562 maintenance_manager.cc:419] P 19f162e164854f2da623a4bb8f9e6bf6: Scheduling FlushMRSOp(de3287296c4547688388388dd4c7e638): perf score=0.033336
I20240502 21:30:50.310866  9426 maintenance_manager.cc:419] P f2fdd596a8b44bbc848fe40f9f07fbe0: Scheduling FlushMRSOp(41ee481a3ece47a0905e93c66dbf949d): perf score=0.033345
W20240502 21:30:50.315829 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:30:50.348956  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 21:30:50.353641  9290 maintenance_manager.cc:419] P 525a093ea5bf40dc9bbf9dc39895d670: Scheduling FlushMRSOp(41ee481a3ece47a0905e93c66dbf949d): perf score=0.033368
I20240502 21:30:50.375478  9154 maintenance_manager.cc:419] P 5543ddab5b484642b139b7c848932a58: Scheduling FlushMRSOp(9240163cfac446d2a14dbd9fcbbf9ff7): perf score=0.033361
W20240502 21:30:50.385658  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:30:50.447153 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:50.449869  9448 maintenance_manager.cc:643] P 19f162e164854f2da623a4bb8f9e6bf6: FlushMRSOp(de3287296c4547688388388dd4c7e638) complete. Timing: real 0.190s	user 0.124s	sys 0.051s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":542,"dirs.queue_time_us":57628,"dirs.run_cpu_time_us":347,"dirs.run_wall_time_us":2252,"drs_written":1,"lbm_read_time_us":129,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1021,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":64613,"threads_started":3,"wal-append.queue_time_us":6658}
I20240502 21:30:50.453498  9562 maintenance_manager.cc:419] P 19f162e164854f2da623a4bb8f9e6bf6: Scheduling FlushMRSOp(9240163cfac446d2a14dbd9fcbbf9ff7): perf score=0.033385
W20240502 21:30:50.531715  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:30:50.551422  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 21:30:50.573773  9448 maintenance_manager.cc:643] P 19f162e164854f2da623a4bb8f9e6bf6: FlushMRSOp(9240163cfac446d2a14dbd9fcbbf9ff7) complete. Timing: real 0.120s	user 0.048s	sys 0.019s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":257,"dirs.run_cpu_time_us":262,"dirs.run_wall_time_us":1846,"drs_written":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1009,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":28212,"threads_started":1,"wal-append.queue_time_us":28357}
I20240502 21:30:50.576182  9562 maintenance_manager.cc:419] P 19f162e164854f2da623a4bb8f9e6bf6: Scheduling UndoDeltaBlockGCOp(de3287296c4547688388388dd4c7e638): 4926 bytes on disk
I20240502 21:30:50.577548  9448 maintenance_manager.cc:643] P 19f162e164854f2da623a4bb8f9e6bf6: UndoDeltaBlockGCOp(de3287296c4547688388388dd4c7e638) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":135,"lbm_reads_lt_1ms":4}
I20240502 21:30:50.579214  9562 maintenance_manager.cc:419] P 19f162e164854f2da623a4bb8f9e6bf6: Scheduling UndoDeltaBlockGCOp(9240163cfac446d2a14dbd9fcbbf9ff7): 4838 bytes on disk
I20240502 21:30:50.579959  9448 maintenance_manager.cc:643] P 19f162e164854f2da623a4bb8f9e6bf6: UndoDeltaBlockGCOp(9240163cfac446d2a14dbd9fcbbf9ff7) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":121,"lbm_reads_lt_1ms":4}
I20240502 21:30:50.713140  9314 maintenance_manager.cc:643] P f2fdd596a8b44bbc848fe40f9f07fbe0: FlushMRSOp(41ee481a3ece47a0905e93c66dbf949d) complete. Timing: real 0.401s	user 0.068s	sys 0.039s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":12887,"dirs.queue_time_us":21639,"dirs.run_cpu_time_us":460,"dirs.run_wall_time_us":7567,"drs_written":1,"lbm_read_time_us":310,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1411,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":80701,"threads_started":3,"wal-append.queue_time_us":44537}
W20240502 21:30:50.724092  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 21:30:50.724751  9426 maintenance_manager.cc:419] P f2fdd596a8b44bbc848fe40f9f07fbe0: Scheduling UndoDeltaBlockGCOp(41ee481a3ece47a0905e93c66dbf949d): 4586 bytes on disk
I20240502 21:30:50.726204  9314 maintenance_manager.cc:643] P f2fdd596a8b44bbc848fe40f9f07fbe0: UndoDeltaBlockGCOp(41ee481a3ece47a0905e93c66dbf949d) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":493,"lbm_reads_lt_1ms":4}
I20240502 21:30:50.797099  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:50.817015  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:50.859689 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:50.876691  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:50.894627 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:50.905377  9042 maintenance_manager.cc:643] P 5543ddab5b484642b139b7c848932a58: FlushMRSOp(9240163cfac446d2a14dbd9fcbbf9ff7) complete. Timing: real 0.529s	user 0.110s	sys 0.017s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":19786,"dirs.queue_time_us":29435,"dirs.run_cpu_time_us":356,"dirs.run_wall_time_us":5072,"drs_written":1,"lbm_read_time_us":135,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1573,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":62094,"threads_started":3,"wal-append.queue_time_us":13366}
I20240502 21:30:50.921844  9154 maintenance_manager.cc:419] P 5543ddab5b484642b139b7c848932a58: Scheduling UndoDeltaBlockGCOp(9240163cfac446d2a14dbd9fcbbf9ff7): 4838 bytes on disk
I20240502 21:30:50.925015  9042 maintenance_manager.cc:643] P 5543ddab5b484642b139b7c848932a58: UndoDeltaBlockGCOp(9240163cfac446d2a14dbd9fcbbf9ff7) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":131,"lbm_reads_lt_1ms":4}
I20240502 21:30:50.928407 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:50.914403  9176 maintenance_manager.cc:643] P 525a093ea5bf40dc9bbf9dc39895d670: FlushMRSOp(41ee481a3ece47a0905e93c66dbf949d) complete. Timing: real 0.560s	user 0.092s	sys 0.028s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":8269,"dirs.queue_time_us":39493,"dirs.run_cpu_time_us":373,"dirs.run_wall_time_us":7891,"drs_written":1,"lbm_read_time_us":137,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1268,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":60859,"threads_started":3,"wal-append.queue_time_us":8900}
I20240502 21:30:50.965385  9290 maintenance_manager.cc:419] P 525a093ea5bf40dc9bbf9dc39895d670: Scheduling UndoDeltaBlockGCOp(41ee481a3ece47a0905e93c66dbf949d): 4586 bytes on disk
I20240502 21:30:50.966953  9176 maintenance_manager.cc:643] P 525a093ea5bf40dc9bbf9dc39895d670: UndoDeltaBlockGCOp(41ee481a3ece47a0905e93c66dbf949d) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":164,"lbm_reads_lt_1ms":4}
W20240502 21:30:50.986233  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:30:50.996568 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:51.017733  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:51.177737  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 21:30:51.179348  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:30:51.403546 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:51.586424  9562 maintenance_manager.cc:419] P 19f162e164854f2da623a4bb8f9e6bf6: Scheduling FlushMRSOp(07d693b680a54e12871121737e3da744): perf score=0.033344
I20240502 21:30:51.622395  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:51.626256  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:51.648515 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:51.661226 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:51.671867  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 21:30:51.683624  9154 maintenance_manager.cc:419] P 5543ddab5b484642b139b7c848932a58: Scheduling FlushMRSOp(4d1a31513bb64578a85adf1f622e5bc0): perf score=0.033374
I20240502 21:30:51.706790  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:51.730473  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:51.743254  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 21:30:51.744341  9290 maintenance_manager.cc:419] P 525a093ea5bf40dc9bbf9dc39895d670: Scheduling FlushMRSOp(58c7004ceb044f5fa9dec54910046b69): perf score=0.033389
I20240502 21:30:51.758780 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:30:51.766638 11295 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.68:40931, user_credentials={real_user=slave}} blocked reactor thread for 43748.8us
W20240502 21:30:51.848814  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 21:30:51.858739  9448 maintenance_manager.cc:643] P 19f162e164854f2da623a4bb8f9e6bf6: FlushMRSOp(07d693b680a54e12871121737e3da744) complete. Timing: real 0.272s	user 0.053s	sys 0.022s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":61237,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":5774,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1054,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":68932,"threads_started":2,"wal-append.queue_time_us":4937}
I20240502 21:30:51.867993  9562 maintenance_manager.cc:419] P 19f162e164854f2da623a4bb8f9e6bf6: Scheduling UndoDeltaBlockGCOp(07d693b680a54e12871121737e3da744): 4584 bytes on disk
I20240502 21:30:51.870251  9448 maintenance_manager.cc:643] P 19f162e164854f2da623a4bb8f9e6bf6: UndoDeltaBlockGCOp(07d693b680a54e12871121737e3da744) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4}
I20240502 21:30:51.871780  9562 maintenance_manager.cc:419] P 19f162e164854f2da623a4bb8f9e6bf6: Scheduling FlushMRSOp(58c7004ceb044f5fa9dec54910046b69): perf score=0.033419
W20240502 21:30:51.938099 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 21:30:51.960328  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:30:51.975766  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:30:51.983059 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:51.988211  9176 maintenance_manager.cc:643] P 525a093ea5bf40dc9bbf9dc39895d670: FlushMRSOp(58c7004ceb044f5fa9dec54910046b69) complete. Timing: real 0.242s	user 0.066s	sys 0.021s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":10348,"dirs.run_cpu_time_us":355,"dirs.run_wall_time_us":6982,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1001,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":46582,"threads_started":2,"wal-append.queue_time_us":36133}
I20240502 21:30:51.991976  9290 maintenance_manager.cc:419] P 525a093ea5bf40dc9bbf9dc39895d670: Scheduling UndoDeltaBlockGCOp(58c7004ceb044f5fa9dec54910046b69): 4925 bytes on disk
I20240502 21:30:51.993717  9176 maintenance_manager.cc:643] P 525a093ea5bf40dc9bbf9dc39895d670: UndoDeltaBlockGCOp(58c7004ceb044f5fa9dec54910046b69) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
I20240502 21:30:52.012773  9042 maintenance_manager.cc:643] P 5543ddab5b484642b139b7c848932a58: FlushMRSOp(4d1a31513bb64578a85adf1f622e5bc0) complete. Timing: real 0.329s	user 0.040s	sys 0.027s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":32042,"dirs.run_cpu_time_us":271,"dirs.run_wall_time_us":4540,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1081,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":46867,"threads_started":2,"wal-append.queue_time_us":12484}
I20240502 21:30:52.014869  9154 maintenance_manager.cc:419] P 5543ddab5b484642b139b7c848932a58: Scheduling UndoDeltaBlockGCOp(4d1a31513bb64578a85adf1f622e5bc0): 4840 bytes on disk
I20240502 21:30:52.016129  9042 maintenance_manager.cc:643] P 5543ddab5b484642b139b7c848932a58: UndoDeltaBlockGCOp(4d1a31513bb64578a85adf1f622e5bc0) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":128,"lbm_reads_lt_1ms":4}
I20240502 21:30:52.111483  9448 maintenance_manager.cc:643] P 19f162e164854f2da623a4bb8f9e6bf6: FlushMRSOp(58c7004ceb044f5fa9dec54910046b69) complete. Timing: real 0.239s	user 0.072s	sys 0.014s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":1457,"drs_written":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1041,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":12634,"threads_started":1,"wal-append.queue_time_us":12458}
I20240502 21:30:52.113276  9562 maintenance_manager.cc:419] P 19f162e164854f2da623a4bb8f9e6bf6: Scheduling UndoDeltaBlockGCOp(58c7004ceb044f5fa9dec54910046b69): 4925 bytes on disk
I20240502 21:30:52.114111  9448 maintenance_manager.cc:643] P 19f162e164854f2da623a4bb8f9e6bf6: UndoDeltaBlockGCOp(58c7004ceb044f5fa9dec54910046b69) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4}
I20240502 21:30:52.320959  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:52.326092  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:30:52.363297 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:52.371948  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:52.434439 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:52.456106  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:52.533157 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:52.541808  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:52.559702 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:52.721879  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 21:30:52.735725  9426 maintenance_manager.cc:419] P f2fdd596a8b44bbc848fe40f9f07fbe0: Scheduling FlushMRSOp(aeee59c5512d448d8f986f959dce1cc8): perf score=0.033351
I20240502 21:30:52.758610  9290 maintenance_manager.cc:419] P 525a093ea5bf40dc9bbf9dc39895d670: Scheduling FlushMRSOp(51936fb1f678458a8795f8246590addb): perf score=0.033351
I20240502 21:30:52.775986  9154 maintenance_manager.cc:419] P 5543ddab5b484642b139b7c848932a58: Scheduling FlushMRSOp(cdf72d25f0f847e5998acbc53979e64d): perf score=0.033358
W20240502 21:30:52.800292  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:30:52.801296 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:52.830642  9314 maintenance_manager.cc:643] P f2fdd596a8b44bbc848fe40f9f07fbe0: FlushMRSOp(aeee59c5512d448d8f986f959dce1cc8) complete. Timing: real 0.095s	user 0.050s	sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":10285,"dirs.run_cpu_time_us":324,"dirs.run_wall_time_us":3129,"drs_written":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1065,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":14229,"threads_started":2,"wal-append.queue_time_us":3955}
I20240502 21:30:52.835202  9426 maintenance_manager.cc:419] P f2fdd596a8b44bbc848fe40f9f07fbe0: Scheduling UndoDeltaBlockGCOp(aeee59c5512d448d8f986f959dce1cc8): 4586 bytes on disk
I20240502 21:30:52.836203  9314 maintenance_manager.cc:643] P f2fdd596a8b44bbc848fe40f9f07fbe0: UndoDeltaBlockGCOp(aeee59c5512d448d8f986f959dce1cc8) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":153,"lbm_reads_lt_1ms":4}
I20240502 21:30:52.838138  9176 maintenance_manager.cc:643] P 525a093ea5bf40dc9bbf9dc39895d670: FlushMRSOp(51936fb1f678458a8795f8246590addb) complete. Timing: real 0.079s	user 0.049s	sys 0.024s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":12700,"dirs.run_cpu_time_us":276,"dirs.run_wall_time_us":3836,"drs_written":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1847,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":19562,"threads_started":2,"wal-append.queue_time_us":6862}
I20240502 21:30:52.839861  9290 maintenance_manager.cc:419] P 525a093ea5bf40dc9bbf9dc39895d670: Scheduling UndoDeltaBlockGCOp(51936fb1f678458a8795f8246590addb): 4925 bytes on disk
I20240502 21:30:52.841019  9176 maintenance_manager.cc:643] P 525a093ea5bf40dc9bbf9dc39895d670: UndoDeltaBlockGCOp(51936fb1f678458a8795f8246590addb) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":139,"lbm_reads_lt_1ms":4}
I20240502 21:30:52.852489  9042 maintenance_manager.cc:643] P 5543ddab5b484642b139b7c848932a58: FlushMRSOp(cdf72d25f0f847e5998acbc53979e64d) complete. Timing: real 0.076s	user 0.049s	sys 0.017s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":8142,"dirs.run_cpu_time_us":387,"dirs.run_wall_time_us":9349,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1311,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":8767,"threads_started":2,"wal-append.queue_time_us":554}
I20240502 21:30:52.856247  9154 maintenance_manager.cc:419] P 5543ddab5b484642b139b7c848932a58: Scheduling UndoDeltaBlockGCOp(cdf72d25f0f847e5998acbc53979e64d): 4840 bytes on disk
I20240502 21:30:52.857327  9042 maintenance_manager.cc:643] P 5543ddab5b484642b139b7c848932a58: UndoDeltaBlockGCOp(cdf72d25f0f847e5998acbc53979e64d) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":143,"lbm_reads_lt_1ms":4}
I20240502 21:30:52.988480  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:53.025811  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:53.034248 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:53.041272  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:53.049170 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:53.060158  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:53.060833  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 21:30:53.076805 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:53.106427 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:30:53.120895  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:30:53.266646 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:53.329037  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:53.336407  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:53.354882 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:53.355129  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:53.405501 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:53.426529 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:53.432513  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:30:53.457623  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 21:30:53.460721  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:53.645504 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:53.663925 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:53.667512  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:53.677313  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:53.696883 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:53.697904  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:53.710076  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:53.738978 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:53.753974  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 21:30:54.036475  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:54.049901  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:30:54.052285  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 21:30:54.053357 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:54.061664  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:54.095571 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:54.096546 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:54.096621  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:54.158042 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:54.387826 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:54.392192  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:54.408231  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:54.417917  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:30:54.439546  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 21:30:54.449044 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:54.449930 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:54.474251  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:54.539091 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:54.570044  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 21:30:54.755084  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:54.772392 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:54.777503 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:54.784204  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:54.787297  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:54.791009 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:54.830111  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:30:55.002710  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:30:55.025197 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:55.085836 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:55.086077 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:55.102651 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:55.103374  9426 maintenance_manager.cc:419] P f2fdd596a8b44bbc848fe40f9f07fbe0: Scheduling FlushMRSOp(4674e376bf7440cda619784491685b44): perf score=0.033390
I20240502 21:30:55.114351  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:55.117486  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:55.120038  9154 maintenance_manager.cc:419] P 5543ddab5b484642b139b7c848932a58: Scheduling FlushMRSOp(8b4486f03fd9474aa0bb2cb4c9149437): perf score=0.033390
I20240502 21:30:55.121585  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:55.129647  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 21:30:55.130388  9562 maintenance_manager.cc:419] P 19f162e164854f2da623a4bb8f9e6bf6: Scheduling FlushMRSOp(8b4486f03fd9474aa0bb2cb4c9149437): perf score=0.033393
I20240502 21:30:55.204490  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:55.236860  9314 maintenance_manager.cc:643] P f2fdd596a8b44bbc848fe40f9f07fbe0: FlushMRSOp(4674e376bf7440cda619784491685b44) complete. Timing: real 0.133s	user 0.056s	sys 0.022s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":21850,"dirs.run_cpu_time_us":275,"dirs.run_wall_time_us":2270,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5271,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":27019,"threads_started":2,"wal-append.queue_time_us":20240}
I20240502 21:30:55.238812  9426 maintenance_manager.cc:419] P f2fdd596a8b44bbc848fe40f9f07fbe0: Scheduling UndoDeltaBlockGCOp(4674e376bf7440cda619784491685b44): 4925 bytes on disk
I20240502 21:30:55.239972  9314 maintenance_manager.cc:643] P f2fdd596a8b44bbc848fe40f9f07fbe0: UndoDeltaBlockGCOp(4674e376bf7440cda619784491685b44) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":263,"lbm_reads_lt_1ms":4}
I20240502 21:30:55.246711  9042 maintenance_manager.cc:643] P 5543ddab5b484642b139b7c848932a58: FlushMRSOp(8b4486f03fd9474aa0bb2cb4c9149437) complete. Timing: real 0.126s	user 0.041s	sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":5000,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":3138,"drs_written":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1151,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":21344,"threads_started":2,"wal-append.queue_time_us":6594}
I20240502 21:30:55.248350  9154 maintenance_manager.cc:419] P 5543ddab5b484642b139b7c848932a58: Scheduling UndoDeltaBlockGCOp(8b4486f03fd9474aa0bb2cb4c9149437): 4586 bytes on disk
I20240502 21:30:55.249152  9042 maintenance_manager.cc:643] P 5543ddab5b484642b139b7c848932a58: UndoDeltaBlockGCOp(8b4486f03fd9474aa0bb2cb4c9149437) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
I20240502 21:30:55.360546  9448 maintenance_manager.cc:643] P 19f162e164854f2da623a4bb8f9e6bf6: FlushMRSOp(8b4486f03fd9474aa0bb2cb4c9149437) complete. Timing: real 0.230s	user 0.079s	sys 0.034s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":9193,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":2417,"drs_written":1,"lbm_read_time_us":133,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1015,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":11734,"threads_started":2,"wal-append.queue_time_us":3186}
I20240502 21:30:55.361907  9562 maintenance_manager.cc:419] P 19f162e164854f2da623a4bb8f9e6bf6: Scheduling FlushMRSOp(1497b781f36c43b39d22a05bb524c59d): perf score=0.033441
I20240502 21:30:55.403779  9448 maintenance_manager.cc:643] P 19f162e164854f2da623a4bb8f9e6bf6: FlushMRSOp(1497b781f36c43b39d22a05bb524c59d) complete. Timing: real 0.042s	user 0.024s	sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":210,"dirs.run_cpu_time_us":315,"dirs.run_wall_time_us":2000,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":775,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":402,"threads_started":1,"wal-append.queue_time_us":578}
I20240502 21:30:55.405608  9562 maintenance_manager.cc:419] P 19f162e164854f2da623a4bb8f9e6bf6: Scheduling UndoDeltaBlockGCOp(1497b781f36c43b39d22a05bb524c59d): 4840 bytes on disk
I20240502 21:30:55.406491  9448 maintenance_manager.cc:643] P 19f162e164854f2da623a4bb8f9e6bf6: UndoDeltaBlockGCOp(1497b781f36c43b39d22a05bb524c59d) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4}
I20240502 21:30:55.408371  9562 maintenance_manager.cc:419] P 19f162e164854f2da623a4bb8f9e6bf6: Scheduling UndoDeltaBlockGCOp(8b4486f03fd9474aa0bb2cb4c9149437): 4586 bytes on disk
I20240502 21:30:55.409410  9448 maintenance_manager.cc:643] P 19f162e164854f2da623a4bb8f9e6bf6: UndoDeltaBlockGCOp(8b4486f03fd9474aa0bb2cb4c9149437) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":165,"lbm_reads_lt_1ms":4}
W20240502 21:30:55.447053 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:30:55.472790 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:30:55.475579  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:30:55.489728  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 21:30:55.493638  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:55.497507 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:55.520951  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:55.525645  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:55.540858 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:55.557062 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:55.563529  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:55.821493 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:55.824111  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:55.824946  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:55.849275  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:30:55.873530  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 21:30:55.875746 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:55.924446 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:55.951001 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:55.965873  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 21:30:55.967784  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:56.025466  9310 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.71:35857, user_credentials={real_user=slave}} blocked reactor thread for 80708.9us
I20240502 21:30:56.108186  9290 maintenance_manager.cc:419] P 525a093ea5bf40dc9bbf9dc39895d670: Scheduling FlushMRSOp(cffcb4f7f7754262beeb06168097f6dc): perf score=0.033342
I20240502 21:30:56.165475  9562 maintenance_manager.cc:419] P 19f162e164854f2da623a4bb8f9e6bf6: Scheduling FlushMRSOp(dc1970b9263d41abb575b9d8639c59ae): perf score=0.033348
I20240502 21:30:56.189718  9176 maintenance_manager.cc:643] P 525a093ea5bf40dc9bbf9dc39895d670: FlushMRSOp(cffcb4f7f7754262beeb06168097f6dc) complete. Timing: real 0.081s	user 0.028s	sys 0.029s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6497,"dirs.run_cpu_time_us":333,"dirs.run_wall_time_us":15964,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1175,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":29664,"threads_started":2,"wal-append.queue_time_us":22807}
I20240502 21:30:56.191395  9290 maintenance_manager.cc:419] P 525a093ea5bf40dc9bbf9dc39895d670: Scheduling UndoDeltaBlockGCOp(cffcb4f7f7754262beeb06168097f6dc): 4586 bytes on disk
I20240502 21:30:56.192368  9176 maintenance_manager.cc:643] P 525a093ea5bf40dc9bbf9dc39895d670: UndoDeltaBlockGCOp(cffcb4f7f7754262beeb06168097f6dc) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
W20240502 21:30:56.205335  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 21:30:56.244529  9448 maintenance_manager.cc:643] P 19f162e164854f2da623a4bb8f9e6bf6: FlushMRSOp(dc1970b9263d41abb575b9d8639c59ae) complete. Timing: real 0.079s	user 0.048s	sys 0.024s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":1395,"dirs.run_cpu_time_us":269,"dirs.run_wall_time_us":5847,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":916,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1587,"threads_started":2,"wal-append.queue_time_us":428}
I20240502 21:30:56.246032  9426 maintenance_manager.cc:419] P f2fdd596a8b44bbc848fe40f9f07fbe0: Scheduling FlushMRSOp(cffcb4f7f7754262beeb06168097f6dc): perf score=0.033372
I20240502 21:30:56.246431  9562 maintenance_manager.cc:419] P 19f162e164854f2da623a4bb8f9e6bf6: Scheduling FlushMRSOp(f0259f01acb249dd99c662151dc21916): perf score=0.033366
I20240502 21:30:56.256160  9154 maintenance_manager.cc:419] P 5543ddab5b484642b139b7c848932a58: Scheduling FlushMRSOp(dc1970b9263d41abb575b9d8639c59ae): perf score=0.033366
I20240502 21:30:56.338871  9314 maintenance_manager.cc:643] P f2fdd596a8b44bbc848fe40f9f07fbe0: FlushMRSOp(cffcb4f7f7754262beeb06168097f6dc) complete. Timing: real 0.093s	user 0.041s	sys 0.023s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6529,"dirs.run_cpu_time_us":275,"dirs.run_wall_time_us":6896,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1088,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":31896,"threads_started":2,"wal-append.queue_time_us":24703}
I20240502 21:30:56.346206  9426 maintenance_manager.cc:419] P f2fdd596a8b44bbc848fe40f9f07fbe0: Scheduling UndoDeltaBlockGCOp(cffcb4f7f7754262beeb06168097f6dc): 4586 bytes on disk
I20240502 21:30:56.347193  9314 maintenance_manager.cc:643] P f2fdd596a8b44bbc848fe40f9f07fbe0: UndoDeltaBlockGCOp(cffcb4f7f7754262beeb06168097f6dc) complete. Timing: real 0.001s	user 0.001s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":134,"lbm_reads_lt_1ms":4}
W20240502 21:30:56.360014  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240502 21:30:56.402236  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:56.408844 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:56.462409  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:30:56.471915  9041 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.66:39071, user_credentials={real_user=slave}} blocked reactor thread for 53403.8us
I20240502 21:30:56.472061 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:56.544036 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:56.544962  9448 maintenance_manager.cc:643] P 19f162e164854f2da623a4bb8f9e6bf6: FlushMRSOp(f0259f01acb249dd99c662151dc21916) complete. Timing: real 0.298s	user 0.046s	sys 0.063s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":175,"dirs.run_cpu_time_us":273,"dirs.run_wall_time_us":1811,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1438,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":132733,"threads_started":1,"wal-append.queue_time_us":131846}
I20240502 21:30:56.556910  9562 maintenance_manager.cc:419] P 19f162e164854f2da623a4bb8f9e6bf6: Scheduling UndoDeltaBlockGCOp(dc1970b9263d41abb575b9d8639c59ae): 4925 bytes on disk
I20240502 21:30:56.558846  9042 maintenance_manager.cc:643] P 5543ddab5b484642b139b7c848932a58: FlushMRSOp(dc1970b9263d41abb575b9d8639c59ae) complete. Timing: real 0.302s	user 0.068s	sys 0.044s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":16146,"dirs.run_cpu_time_us":249,"dirs.run_wall_time_us":2539,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1007,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":64726,"threads_started":2,"wal-append.queue_time_us":54962}
I20240502 21:30:56.559971  9448 maintenance_manager.cc:643] P 19f162e164854f2da623a4bb8f9e6bf6: UndoDeltaBlockGCOp(dc1970b9263d41abb575b9d8639c59ae) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
I20240502 21:30:56.561412  9154 maintenance_manager.cc:419] P 5543ddab5b484642b139b7c848932a58: Scheduling UndoDeltaBlockGCOp(dc1970b9263d41abb575b9d8639c59ae): 4925 bytes on disk
I20240502 21:30:56.562271  9042 maintenance_manager.cc:643] P 5543ddab5b484642b139b7c848932a58: UndoDeltaBlockGCOp(dc1970b9263d41abb575b9d8639c59ae) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20240502 21:30:56.567993 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:56.571350  9562 maintenance_manager.cc:419] P 19f162e164854f2da623a4bb8f9e6bf6: Scheduling UndoDeltaBlockGCOp(f0259f01acb249dd99c662151dc21916): 4840 bytes on disk
I20240502 21:30:56.579720  9448 maintenance_manager.cc:643] P 19f162e164854f2da623a4bb8f9e6bf6: UndoDeltaBlockGCOp(f0259f01acb249dd99c662151dc21916) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":143,"lbm_reads_lt_1ms":4}
I20240502 21:30:56.629577  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:56.630216  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:56.679584  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 21:30:56.817126  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:56.828174  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:56.837312  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:56.850701 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:30:56.856331  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 21:30:56.871476 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:56.880681 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:56.893433  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:56.914454  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:30:56.965780 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:30:56.980612  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 21:30:56.982344  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:30:57.080616 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:57.200146  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:57.201315 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:57.208220  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:57.231528  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:57.237579 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:57.244501  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:57.276459 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:57.319255  9154 maintenance_manager.cc:419] P 5543ddab5b484642b139b7c848932a58: Scheduling FlushMRSOp(32c306d724954a16be21a34aa4cc0b5f): perf score=0.033353
I20240502 21:30:57.357767  9426 maintenance_manager.cc:419] P f2fdd596a8b44bbc848fe40f9f07fbe0: Scheduling FlushMRSOp(8a2e06253c51405fa9b6bc9e87844760): perf score=0.033362
I20240502 21:30:57.415244  9042 maintenance_manager.cc:643] P 5543ddab5b484642b139b7c848932a58: FlushMRSOp(32c306d724954a16be21a34aa4cc0b5f) complete. Timing: real 0.096s	user 0.059s	sys 0.027s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":28736,"dirs.run_cpu_time_us":326,"dirs.run_wall_time_us":1943,"drs_written":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1155,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":28964,"threads_started":2,"wal-append.queue_time_us":579}
I20240502 21:30:57.417068  9154 maintenance_manager.cc:419] P 5543ddab5b484642b139b7c848932a58: Scheduling UndoDeltaBlockGCOp(32c306d724954a16be21a34aa4cc0b5f): 4840 bytes on disk
I20240502 21:30:57.417877  9042 maintenance_manager.cc:643] P 5543ddab5b484642b139b7c848932a58: UndoDeltaBlockGCOp(32c306d724954a16be21a34aa4cc0b5f) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":136,"lbm_reads_lt_1ms":4}
I20240502 21:30:57.435415  9314 maintenance_manager.cc:643] P f2fdd596a8b44bbc848fe40f9f07fbe0: FlushMRSOp(8a2e06253c51405fa9b6bc9e87844760) complete. Timing: real 0.077s	user 0.055s	sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":660,"dirs.run_cpu_time_us":291,"dirs.run_wall_time_us":5027,"drs_written":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1114,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":638,"threads_started":2,"wal-append.queue_time_us":644}
I20240502 21:30:57.437691  9426 maintenance_manager.cc:419] P f2fdd596a8b44bbc848fe40f9f07fbe0: Scheduling UndoDeltaBlockGCOp(8a2e06253c51405fa9b6bc9e87844760): 4925 bytes on disk
I20240502 21:30:57.438879  9314 maintenance_manager.cc:643] P f2fdd596a8b44bbc848fe40f9f07fbe0: UndoDeltaBlockGCOp(8a2e06253c51405fa9b6bc9e87844760) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":201,"lbm_reads_lt_1ms":4}
I20240502 21:30:57.532939 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:57.534725  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:57.544735  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 21:30:57.549367  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:30:57.560523  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 21:30:57.563263 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:57.578886  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:30:57.587038 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:57.596154 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:57.681815  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:57.708851  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:57.907423  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 21:30:57.909202  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:57.930665 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:57.950727  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:57.954278 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:57.954716  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:57.957208  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:57.994760 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 21:30:58.013100 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:58.131196  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:30:58.134424 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:58.240411  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:58.244418 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:58.254207  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:58.274041  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:58.278693 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:58.290854  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:58.295482 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:58.314826  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:30:58.397779  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 21:30:58.552026  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:58.553103  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:58.558385 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:58.571589 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:58.573007 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:58.577294 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:58.599372  9562 maintenance_manager.cc:419] P 19f162e164854f2da623a4bb8f9e6bf6: Scheduling FlushMRSOp(be34ec89876d42b69efe1878c0d9bc19): perf score=0.033339
I20240502 21:30:58.618491  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:58.625274  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:30:58.693614  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 21:30:58.707290  9290 maintenance_manager.cc:419] P 525a093ea5bf40dc9bbf9dc39895d670: Scheduling FlushMRSOp(f9ae8fac693d4d68b6741f2509e1265e): perf score=0.033359
I20240502 21:30:58.733116  9448 maintenance_manager.cc:643] P 19f162e164854f2da623a4bb8f9e6bf6: FlushMRSOp(be34ec89876d42b69efe1878c0d9bc19) complete. Timing: real 0.133s	user 0.048s	sys 0.028s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":24831,"dirs.run_cpu_time_us":255,"dirs.run_wall_time_us":1866,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1317,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":31354,"threads_started":2,"wal-append.queue_time_us":6483}
I20240502 21:30:58.734723  9562 maintenance_manager.cc:419] P 19f162e164854f2da623a4bb8f9e6bf6: Scheduling UndoDeltaBlockGCOp(be34ec89876d42b69efe1878c0d9bc19): 4586 bytes on disk
I20240502 21:30:58.735478  9448 maintenance_manager.cc:643] P 19f162e164854f2da623a4bb8f9e6bf6: UndoDeltaBlockGCOp(be34ec89876d42b69efe1878c0d9bc19) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
I20240502 21:30:58.803942  9176 maintenance_manager.cc:643] P 525a093ea5bf40dc9bbf9dc39895d670: FlushMRSOp(f9ae8fac693d4d68b6741f2509e1265e) complete. Timing: real 0.096s	user 0.062s	sys 0.020s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3301,"dirs.run_cpu_time_us":272,"dirs.run_wall_time_us":1991,"drs_written":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4,"lbm_write_time_us":844,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":1168,"threads_started":2,"wal-append.queue_time_us":455}
I20240502 21:30:58.805368  9290 maintenance_manager.cc:419] P 525a093ea5bf40dc9bbf9dc39895d670: Scheduling UndoDeltaBlockGCOp(f9ae8fac693d4d68b6741f2509e1265e): 4838 bytes on disk
I20240502 21:30:58.807013  9176 maintenance_manager.cc:643] P 525a093ea5bf40dc9bbf9dc39895d670: UndoDeltaBlockGCOp(f9ae8fac693d4d68b6741f2509e1265e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":135,"lbm_reads_lt_1ms":4}
I20240502 21:30:58.935889  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:58.956040  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:58.963068  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:58.963583 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:58.964184 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:58.964368  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:58.973896 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:30:59.056633  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:30:59.128458 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:59.264532  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:59.271651 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:59.284704  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:59.288669 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:59.289920  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:59.309248 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:59.314886  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:59.322122  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:30:59.370867  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:30:59.505249 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:59.551009 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:59.563864  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:59.585186  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:59.610680  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:30:59.649071 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:59.650841 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:30:59.713037  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:30:59.929283 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:30:59.929855  9154 maintenance_manager.cc:419] P 5543ddab5b484642b139b7c848932a58: Scheduling FlushMRSOp(79dedaa327b6479e8325f91481c1688e): perf score=0.033357
I20240502 21:30:59.951593  9426 maintenance_manager.cc:419] P f2fdd596a8b44bbc848fe40f9f07fbe0: Scheduling FlushMRSOp(87f35aff2b104c01a9d9d48a7e9c9b23): perf score=0.033371
I20240502 21:30:59.968288 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:30:59.975067  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:30:59.975723  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:30:59.993440  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:00.020220 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:00.038403  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:00.053748 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:00.085147  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:31:00.090646  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 21:31:00.133405  9042 maintenance_manager.cc:643] P 5543ddab5b484642b139b7c848932a58: FlushMRSOp(79dedaa327b6479e8325f91481c1688e) complete. Timing: real 0.203s	user 0.096s	sys 0.051s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":33887,"dirs.run_cpu_time_us":349,"dirs.run_wall_time_us":3520,"drs_written":1,"lbm_read_time_us":201,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1491,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":125652,"threads_started":2,"wal-append.queue_time_us":91681}
I20240502 21:31:00.135425  9154 maintenance_manager.cc:419] P 5543ddab5b484642b139b7c848932a58: Scheduling UndoDeltaBlockGCOp(79dedaa327b6479e8325f91481c1688e): 4586 bytes on disk
I20240502 21:31:00.155781  9042 maintenance_manager.cc:643] P 5543ddab5b484642b139b7c848932a58: UndoDeltaBlockGCOp(79dedaa327b6479e8325f91481c1688e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
I20240502 21:31:00.157200  9154 maintenance_manager.cc:419] P 5543ddab5b484642b139b7c848932a58: Scheduling FlushMRSOp(94d2cdf8ae6941668b5f680c558656e8): perf score=0.033413
I20240502 21:31:00.292829  9042 maintenance_manager.cc:643] P 5543ddab5b484642b139b7c848932a58: FlushMRSOp(94d2cdf8ae6941668b5f680c558656e8) complete. Timing: real 0.135s	user 0.047s	sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":65,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":1619,"drs_written":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4,"lbm_write_time_us":983,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":14677,"threads_started":1,"wal-append.queue_time_us":16025}
I20240502 21:31:00.294363  9154 maintenance_manager.cc:419] P 5543ddab5b484642b139b7c848932a58: Scheduling UndoDeltaBlockGCOp(94d2cdf8ae6941668b5f680c558656e8): 4925 bytes on disk
I20240502 21:31:00.295198  9042 maintenance_manager.cc:643] P 5543ddab5b484642b139b7c848932a58: UndoDeltaBlockGCOp(94d2cdf8ae6941668b5f680c558656e8) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":123,"lbm_reads_lt_1ms":4}
I20240502 21:31:00.297806  9314 maintenance_manager.cc:643] P f2fdd596a8b44bbc848fe40f9f07fbe0: FlushMRSOp(87f35aff2b104c01a9d9d48a7e9c9b23) complete. Timing: real 0.346s	user 0.083s	sys 0.011s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3350,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1579,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1114,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":18827,"threads_started":2,"wal-append.queue_time_us":25854}
I20240502 21:31:00.299403  9426 maintenance_manager.cc:419] P f2fdd596a8b44bbc848fe40f9f07fbe0: Scheduling UndoDeltaBlockGCOp(87f35aff2b104c01a9d9d48a7e9c9b23): 4840 bytes on disk
I20240502 21:31:00.304339  9314 maintenance_manager.cc:643] P f2fdd596a8b44bbc848fe40f9f07fbe0: UndoDeltaBlockGCOp(87f35aff2b104c01a9d9d48a7e9c9b23) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":155,"lbm_reads_lt_1ms":4}
I20240502 21:31:00.308028  9426 maintenance_manager.cc:419] P f2fdd596a8b44bbc848fe40f9f07fbe0: Scheduling FlushMRSOp(79dedaa327b6479e8325f91481c1688e): perf score=0.033458
W20240502 21:31:00.340075  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 21:31:00.358959  9314 maintenance_manager.cc:643] P f2fdd596a8b44bbc848fe40f9f07fbe0: FlushMRSOp(79dedaa327b6479e8325f91481c1688e) complete. Timing: real 0.051s	user 0.015s	sys 0.024s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":92,"dirs.run_cpu_time_us":269,"dirs.run_wall_time_us":1953,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":773,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":3968,"thread_start_us":322,"threads_started":1,"wal-append.queue_time_us":434}
I20240502 21:31:00.360165  9426 maintenance_manager.cc:419] P f2fdd596a8b44bbc848fe40f9f07fbe0: Scheduling UndoDeltaBlockGCOp(79dedaa327b6479e8325f91481c1688e): 4586 bytes on disk
I20240502 21:31:00.360780  9314 maintenance_manager.cc:643] P f2fdd596a8b44bbc848fe40f9f07fbe0: UndoDeltaBlockGCOp(79dedaa327b6479e8325f91481c1688e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240502 21:31:00.434818  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:00.439486  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:00.461122 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:00.484711 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:00.491878  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:00.492758 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:00.497519  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:00.527848 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:31:00.535081 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:00.761561 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:31:00.776309  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 21:31:00.783907  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:00.794888  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:00.845796  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:00.847340 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:00.874117 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:00.943104 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:00.962996  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:00.979336  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 21:31:00.998605  9562 maintenance_manager.cc:419] P 19f162e164854f2da623a4bb8f9e6bf6: Scheduling FlushMRSOp(c56ac926025e47218e9df4088184a278): perf score=0.033346
I20240502 21:31:01.051236  9154 maintenance_manager.cc:419] P 5543ddab5b484642b139b7c848932a58: Scheduling FlushMRSOp(e46065d94e874293a836bc74241b8e06): perf score=0.033356
I20240502 21:31:01.068282  9290 maintenance_manager.cc:419] P 525a093ea5bf40dc9bbf9dc39895d670: Scheduling FlushMRSOp(c56ac926025e47218e9df4088184a278): perf score=0.033353
I20240502 21:31:01.074357  9448 maintenance_manager.cc:643] P 19f162e164854f2da623a4bb8f9e6bf6: FlushMRSOp(c56ac926025e47218e9df4088184a278) complete. Timing: real 0.075s	user 0.041s	sys 0.010s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3776,"dirs.run_cpu_time_us":288,"dirs.run_wall_time_us":1993,"drs_written":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1195,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":5386,"threads_started":2,"wal-append.queue_time_us":446}
I20240502 21:31:01.076098  9562 maintenance_manager.cc:419] P 19f162e164854f2da623a4bb8f9e6bf6: Scheduling FlushMRSOp(a2f3edaf1c874cefb820f027f90d1185): perf score=0.033361
W20240502 21:31:01.098306  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 21:31:01.115128  9426 maintenance_manager.cc:419] P f2fdd596a8b44bbc848fe40f9f07fbe0: Scheduling FlushMRSOp(a2f3edaf1c874cefb820f027f90d1185): perf score=0.033364
W20240502 21:31:01.216588  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 21:31:01.223222  9042 maintenance_manager.cc:643] P 5543ddab5b484642b139b7c848932a58: FlushMRSOp(e46065d94e874293a836bc74241b8e06) complete. Timing: real 0.172s	user 0.081s	sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6514,"dirs.run_cpu_time_us":394,"dirs.run_wall_time_us":10396,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1150,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":40288,"threads_started":2,"wal-append.queue_time_us":33478}
I20240502 21:31:01.225744  9154 maintenance_manager.cc:419] P 5543ddab5b484642b139b7c848932a58: Scheduling UndoDeltaBlockGCOp(e46065d94e874293a836bc74241b8e06): 4586 bytes on disk
I20240502 21:31:01.226709  9042 maintenance_manager.cc:643] P 5543ddab5b484642b139b7c848932a58: UndoDeltaBlockGCOp(e46065d94e874293a836bc74241b8e06) complete. Timing: real 0.001s	user 0.001s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":138,"lbm_reads_lt_1ms":4}
I20240502 21:31:01.269223  9314 maintenance_manager.cc:643] P f2fdd596a8b44bbc848fe40f9f07fbe0: FlushMRSOp(a2f3edaf1c874cefb820f027f90d1185) complete. Timing: real 0.154s	user 0.058s	sys 0.018s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":5206,"dirs.run_cpu_time_us":335,"dirs.run_wall_time_us":7117,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":10004,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":17028,"threads_started":2,"wal-append.queue_time_us":11560}
I20240502 21:31:01.288758  9176 maintenance_manager.cc:643] P 525a093ea5bf40dc9bbf9dc39895d670: FlushMRSOp(c56ac926025e47218e9df4088184a278) complete. Timing: real 0.220s	user 0.071s	sys 0.037s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":32806,"dirs.run_cpu_time_us":255,"dirs.run_wall_time_us":7306,"drs_written":1,"lbm_read_time_us":133,"lbm_reads_lt_1ms":4,"lbm_write_time_us":12801,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":29901,"threads_started":2,"wal-append.queue_time_us":1643}
I20240502 21:31:01.290977  9290 maintenance_manager.cc:419] P 525a093ea5bf40dc9bbf9dc39895d670: Scheduling UndoDeltaBlockGCOp(c56ac926025e47218e9df4088184a278): 4925 bytes on disk
I20240502 21:31:01.292243  9176 maintenance_manager.cc:643] P 525a093ea5bf40dc9bbf9dc39895d670: UndoDeltaBlockGCOp(c56ac926025e47218e9df4088184a278) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":140,"lbm_reads_lt_1ms":4}
I20240502 21:31:01.294041  9290 maintenance_manager.cc:419] P 525a093ea5bf40dc9bbf9dc39895d670: Scheduling FlushMRSOp(e46065d94e874293a836bc74241b8e06): perf score=0.033412
I20240502 21:31:01.301126  9448 maintenance_manager.cc:643] P 19f162e164854f2da623a4bb8f9e6bf6: FlushMRSOp(a2f3edaf1c874cefb820f027f90d1185) complete. Timing: real 0.214s	user 0.057s	sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":115,"dirs.run_cpu_time_us":389,"dirs.run_wall_time_us":1842,"drs_written":1,"lbm_read_time_us":131,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1359,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":13127552,"thread_start_us":7104,"threads_started":1,"wal-append.queue_time_us":6910}
I20240502 21:31:01.302690  9562 maintenance_manager.cc:419] P 19f162e164854f2da623a4bb8f9e6bf6: Scheduling UndoDeltaBlockGCOp(c56ac926025e47218e9df4088184a278): 4925 bytes on disk
I20240502 21:31:01.308065  9448 maintenance_manager.cc:643] P 19f162e164854f2da623a4bb8f9e6bf6: UndoDeltaBlockGCOp(c56ac926025e47218e9df4088184a278) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20240502 21:31:01.309263  9426 maintenance_manager.cc:419] P f2fdd596a8b44bbc848fe40f9f07fbe0: Scheduling UndoDeltaBlockGCOp(a2f3edaf1c874cefb820f027f90d1185): 4841 bytes on disk
I20240502 21:31:01.309465  9562 maintenance_manager.cc:419] P 19f162e164854f2da623a4bb8f9e6bf6: Scheduling UndoDeltaBlockGCOp(a2f3edaf1c874cefb820f027f90d1185): 4841 bytes on disk
I20240502 21:31:01.311210  9448 maintenance_manager.cc:643] P 19f162e164854f2da623a4bb8f9e6bf6: UndoDeltaBlockGCOp(a2f3edaf1c874cefb820f027f90d1185) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240502 21:31:01.313251  9314 maintenance_manager.cc:643] P f2fdd596a8b44bbc848fe40f9f07fbe0: UndoDeltaBlockGCOp(a2f3edaf1c874cefb820f027f90d1185) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240502 21:31:01.384466  9176 maintenance_manager.cc:643] P 525a093ea5bf40dc9bbf9dc39895d670: FlushMRSOp(e46065d94e874293a836bc74241b8e06) complete. Timing: real 0.067s	user 0.043s	sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":69,"dirs.run_cpu_time_us":304,"dirs.run_wall_time_us":1816,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1082,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":3494,"threads_started":1,"wal-append.queue_time_us":5086}
I20240502 21:31:01.385957  9290 maintenance_manager.cc:419] P 525a093ea5bf40dc9bbf9dc39895d670: Scheduling UndoDeltaBlockGCOp(e46065d94e874293a836bc74241b8e06): 4586 bytes on disk
I20240502 21:31:01.387199  9176 maintenance_manager.cc:643] P 525a093ea5bf40dc9bbf9dc39895d670: UndoDeltaBlockGCOp(e46065d94e874293a836bc74241b8e06) complete. Timing: real 0.001s	user 0.000s	sys 0.001s Metrics: {"cfile_init":1,"lbm_read_time_us":162,"lbm_reads_lt_1ms":4}
W20240502 21:31:01.398224 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:01.404423  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 21:31:01.428625  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:01.431962 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:01.454160  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:01.459214  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:01.563455 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:01.567080 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:01.580998  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:01.648012  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 21:31:01.788674  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:01.798133 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:01.801491 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:01.807184  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:01.811787  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:01.823745  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:01.843365 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:01.873176 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:01.949429  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:31:01.980654 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:31:01.994195  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 21:31:02.109740  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:02.129617 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:02.134608  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:02.144801  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:02.150542  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 21:31:02.151716  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 21:31:02.164564  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:02.183617 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:02.196172 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:02.393682  9290 maintenance_manager.cc:419] P 525a093ea5bf40dc9bbf9dc39895d670: Scheduling FlushMRSOp(8129d13163f64368b731ba4eaeaaa81c): perf score=0.033373
W20240502 21:31:02.421876 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:02.439404  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 21:31:02.473111  9176 maintenance_manager.cc:643] P 525a093ea5bf40dc9bbf9dc39895d670: FlushMRSOp(8129d13163f64368b731ba4eaeaaa81c) complete. Timing: real 0.079s	user 0.041s	sys 0.028s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":754,"dirs.run_cpu_time_us":347,"dirs.run_wall_time_us":2120,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2581,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1212,"threads_started":2,"wal-append.queue_time_us":2836}
I20240502 21:31:02.475572  9290 maintenance_manager.cc:419] P 525a093ea5bf40dc9bbf9dc39895d670: Scheduling UndoDeltaBlockGCOp(8129d13163f64368b731ba4eaeaaa81c): 4584 bytes on disk
I20240502 21:31:02.477186  9176 maintenance_manager.cc:643] P 525a093ea5bf40dc9bbf9dc39895d670: UndoDeltaBlockGCOp(8129d13163f64368b731ba4eaeaaa81c) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":133,"lbm_reads_lt_1ms":4}
I20240502 21:31:02.479100  9290 maintenance_manager.cc:419] P 525a093ea5bf40dc9bbf9dc39895d670: Scheduling FlushMRSOp(3caa36212ba444aa88d153ff9f90122e): perf score=0.033388
I20240502 21:31:02.492998  9154 maintenance_manager.cc:419] P 5543ddab5b484642b139b7c848932a58: Scheduling FlushMRSOp(3d1a5050d3f944969a560994a23700ad): perf score=0.033386
I20240502 21:31:02.527290 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:02.533139  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:02.552927  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:02.565094  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:02.570565  9562 maintenance_manager.cc:419] P 19f162e164854f2da623a4bb8f9e6bf6: Scheduling FlushMRSOp(3d1a5050d3f944969a560994a23700ad): perf score=0.033402
I20240502 21:31:02.576160  9426 maintenance_manager.cc:419] P f2fdd596a8b44bbc848fe40f9f07fbe0: Scheduling FlushMRSOp(3caa36212ba444aa88d153ff9f90122e): perf score=0.033402
I20240502 21:31:02.619575 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:02.641286 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:02.698346  9176 maintenance_manager.cc:643] P 525a093ea5bf40dc9bbf9dc39895d670: FlushMRSOp(3caa36212ba444aa88d153ff9f90122e) complete. Timing: real 0.219s	user 0.106s	sys 0.074s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":94,"dirs.run_cpu_time_us":291,"dirs.run_wall_time_us":2093,"drs_written":1,"lbm_read_time_us":2881,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":1202,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":124987,"threads_started":1,"wal-append.queue_time_us":125135}
I20240502 21:31:02.708778  9290 maintenance_manager.cc:419] P 525a093ea5bf40dc9bbf9dc39895d670: Scheduling UndoDeltaBlockGCOp(3caa36212ba444aa88d153ff9f90122e): 4923 bytes on disk
I20240502 21:31:02.709625  9176 maintenance_manager.cc:643] P 525a093ea5bf40dc9bbf9dc39895d670: UndoDeltaBlockGCOp(3caa36212ba444aa88d153ff9f90122e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
I20240502 21:31:02.767997  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:02.769428  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 21:31:02.807377  9042 maintenance_manager.cc:643] P 5543ddab5b484642b139b7c848932a58: FlushMRSOp(3d1a5050d3f944969a560994a23700ad) complete. Timing: real 0.314s	user 0.059s	sys 0.006s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":17401,"dirs.run_cpu_time_us":430,"dirs.run_wall_time_us":33513,"drs_written":1,"lbm_read_time_us":131,"lbm_reads_lt_1ms":4,"lbm_write_time_us":8844,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":676,"thread_start_us":27201,"threads_started":2,"wal-append.queue_time_us":9849}
I20240502 21:31:02.809170  9154 maintenance_manager.cc:419] P 5543ddab5b484642b139b7c848932a58: Scheduling UndoDeltaBlockGCOp(3d1a5050d3f944969a560994a23700ad): 4838 bytes on disk
I20240502 21:31:02.812659  9042 maintenance_manager.cc:643] P 5543ddab5b484642b139b7c848932a58: UndoDeltaBlockGCOp(3d1a5050d3f944969a560994a23700ad) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":135,"lbm_reads_lt_1ms":4}
I20240502 21:31:02.849131  9314 maintenance_manager.cc:643] P f2fdd596a8b44bbc848fe40f9f07fbe0: FlushMRSOp(3caa36212ba444aa88d153ff9f90122e) complete. Timing: real 0.273s	user 0.060s	sys 0.013s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":17451,"dirs.run_cpu_time_us":347,"dirs.run_wall_time_us":4965,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1186,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":34194,"threads_started":2,"wal-append.queue_time_us":16030}
I20240502 21:31:02.851783  9426 maintenance_manager.cc:419] P f2fdd596a8b44bbc848fe40f9f07fbe0: Scheduling FlushMRSOp(8129d13163f64368b731ba4eaeaaa81c): perf score=0.033476
W20240502 21:31:02.858183 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:02.869236  9448 maintenance_manager.cc:643] P 19f162e164854f2da623a4bb8f9e6bf6: FlushMRSOp(3d1a5050d3f944969a560994a23700ad) complete. Timing: real 0.298s	user 0.056s	sys 0.017s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":7890,"dirs.run_cpu_time_us":395,"dirs.run_wall_time_us":7253,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":926,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":53069,"threads_started":2,"wal-append.queue_time_us":14057}
I20240502 21:31:02.872272  9562 maintenance_manager.cc:419] P 19f162e164854f2da623a4bb8f9e6bf6: Scheduling UndoDeltaBlockGCOp(3d1a5050d3f944969a560994a23700ad): 4838 bytes on disk
I20240502 21:31:02.873137  9448 maintenance_manager.cc:643] P 19f162e164854f2da623a4bb8f9e6bf6: UndoDeltaBlockGCOp(3d1a5050d3f944969a560994a23700ad) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":129,"lbm_reads_lt_1ms":4}
I20240502 21:31:02.949582  9314 maintenance_manager.cc:643] P f2fdd596a8b44bbc848fe40f9f07fbe0: FlushMRSOp(8129d13163f64368b731ba4eaeaaa81c) complete. Timing: real 0.097s	user 0.063s	sys 0.020s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":86,"dirs.run_cpu_time_us":421,"dirs.run_wall_time_us":2253,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1014,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":586,"threads_started":1,"wal-append.queue_time_us":816}
I20240502 21:31:02.951268  9426 maintenance_manager.cc:419] P f2fdd596a8b44bbc848fe40f9f07fbe0: Scheduling UndoDeltaBlockGCOp(3caa36212ba444aa88d153ff9f90122e): 4923 bytes on disk
W20240502 21:31:02.956496 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 21:31:02.956825  9314 maintenance_manager.cc:643] P f2fdd596a8b44bbc848fe40f9f07fbe0: UndoDeltaBlockGCOp(3caa36212ba444aa88d153ff9f90122e) complete. Timing: real 0.002s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":1063,"lbm_reads_lt_1ms":4}
I20240502 21:31:02.958124  9426 maintenance_manager.cc:419] P f2fdd596a8b44bbc848fe40f9f07fbe0: Scheduling UndoDeltaBlockGCOp(8129d13163f64368b731ba4eaeaaa81c): 4584 bytes on disk
I20240502 21:31:02.958834  9314 maintenance_manager.cc:643] P f2fdd596a8b44bbc848fe40f9f07fbe0: UndoDeltaBlockGCOp(8129d13163f64368b731ba4eaeaaa81c) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
I20240502 21:31:03.049249 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:03.057526 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:03.057586  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:03.058140  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:03.082388 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:03.108577  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:03.108597  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:03.153904  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:31:03.193555  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:31:03.277031  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:31:03.368856 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:03.390185  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:03.392189 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:03.409162  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:31:03.416687  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 21:31:03.422041  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:03.436422  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:03.439554 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:03.441866 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:03.594966  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:31:03.614264  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 21:31:03.723548  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:03.728960 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:03.737072  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:03.753814  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:03.755998 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:03.776103 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:03.787496  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:03.883625 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:04.016755  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 21:31:04.062558  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:04.065182 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:04.068516  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:04.088799 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:04.093248  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:04.096395 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:04.115584  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:04.380993 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:04.390946 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:04.404282  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:04.410926  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:04.418709 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:04.435546  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:04.446600  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:04.470134 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:04.705776  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:31:04.713841  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 21:31:04.752558 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:04.773006  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:04.796388  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:04.809940 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:04.821784 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:04.833554  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:04.839962 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:04.849766  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:04.920069  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:31:04.935789  9313 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.72:40467, user_credentials={real_user=slave}} blocked reactor thread for 49655.1us
W20240502 21:31:05.165063  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 21:31:05.167559  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:05.186131 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:05.188483  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:05.215783  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:05.216943 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:05.267747 25065 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:31:05.268034 25065 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:31:05.269017 25065 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:31:05.272213 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:05.275347 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
I20240502 21:31:05.275550 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:31:05.276463 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:05.277074 25065 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 21:31:05.279801  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:05.428385 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:05.517390  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:05.530573  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:05.541007 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:05.543915 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:05.544834 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:05.564309  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:05.592568  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:05.601729 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:31:05.624317  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 21:31:05.845070  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:05.854847  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:05.867319 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:31:05.873638  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 21:31:05.883940  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:05.888557  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:05.893394 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:05.898589 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:05.946513  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:31:05.958451 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:05.969386  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:31:06.055713  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 21:31:06.176010  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:06.204447 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:06.209298  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:06.231492  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:06.238983 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:06.248510 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:06.249575  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:06.451478  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 21:31:06.452793  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 21:31:06.536393  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:06.548547  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:06.566256  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:06.597990  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:06.600708 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:06.603030 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:06.632043  9173 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 63562.4us
I20240502 21:31:06.636142 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:31:06.660878 11430 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 106405us
W20240502 21:31:06.662179 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:06.933735  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:06.934178 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:06.939929  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:06.950861 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:06.956104  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:06.960785  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:06.973241 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:06.980736 25065 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:31:06.980965 25065 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:31:06.981732 25065 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:31:06.983726 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:31:06.985167 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:06.986243 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
I20240502 21:31:06.986380 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:31:06.987322 25065 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 21:31:07.059087 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:07.088197  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:31:07.139318  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:31:07.234196  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 21:31:07.259477 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:07.262560 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:07.263813  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:07.267019  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:07.271425 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:07.273618  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:07.291718  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:07.353947  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:31:07.435734  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 21:31:07.436842  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:31:07.545380 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:07.587735  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:07.587719 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:07.593326  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:07.610525  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:07.625485 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:07.657740  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:07.670107 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:07.926621 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:07.937433  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:07.940093  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:07.965417 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:07.965814 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:07.971619  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:07.999172  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:08.037299  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:31:08.158185 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:08.267504  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:08.267580  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:08.268096 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 21:31:08.288478  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:08.320075 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:08.335462  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:08.341480 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:08.354260 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:08.571897  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:08.586804 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:31:08.599922  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:31:08.623827 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:08.636739  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:08.660622  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:08.693694  9041 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.72:40467, user_credentials={real_user=slave}} blocked reactor thread for 109780us
W20240502 21:31:08.694537  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:31:08.704110  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:31:08.717845  9173 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 117928us
W20240502 21:31:08.718786  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 21:31:08.731027  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:08.731536 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:08.731945 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:08.803675  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 21:31:08.997032  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:08.999825  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:09.006636 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:09.032482  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:09.032868 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:09.034386  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:09.045332 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:09.108943 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:09.139072  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 21:31:09.321224  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:09.342605 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:09.344905  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:09.358892 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:09.374457  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:09.376528 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:09.430354  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:09.655854  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:09.670912 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:09.679416  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:31:09.690735  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 21:31:09.691545  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:09.697926  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:09.704787 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:09.710336 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:09.711443 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:09.726214  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:31:09.963210  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 21:31:09.996742  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:10.002183  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:10.013494 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:10.027243  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:10.028625 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:10.033257  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:10.040545 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:10.143296 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:10.350603  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:10.363813 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:10.379874  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:10.380250  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:10.391127 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:10.392246  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:10.393350 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:10.406358  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:31:10.494617  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:31:10.564066 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:10.659706  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:10.668861  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:10.672773  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:10.677742 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:10.690692 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:10.700907 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:10.711009  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:10.740908 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 21:31:10.985782  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:10.987726  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:11.005599  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:11.012820 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:11.018807 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:11.021152 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:11.025696 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:11.036639  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:11.071982  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:31:11.270476  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 21:31:11.286298 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:11.288832  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:11.298427  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:11.316300 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:11.326707  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:11.337060  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:11.341419 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:11.373347  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:31:11.409026  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:31:11.511837 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:11.524190  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 21:31:11.525329  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 21:31:11.614357  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:11.625918  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:11.630101  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:11.650046 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:11.722656 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:11.731873  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:11.749327 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:11.771919  8946 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:11.780671  8946 catalog_manager.cc:4573] TS 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4d1a31513bb64578a85adf1f622e5bc0: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:11.904930 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:12.001430  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:12.024971 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:12.038028 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:12.040068  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:12.041433 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:12.043244  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:12.059419  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:12.147805  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:31:12.185616  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:31:12.249305 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 21:31:12.343761 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:12.380307  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 21:31:12.386535 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:12.390856 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:12.394064  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:12.396566  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:12.396677  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:12.418184  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:31:12.435820  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 27 similar messages]
I20240502 21:31:12.440877 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:12.462379  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 21:31:12.696511  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:12.721541 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:12.725566 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:12.728933  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:12.732982 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:12.739619  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:12.748001  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:31:12.801358 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:12.972182  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 21:31:13.051930 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:13.073571 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:13.074473  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:13.085367  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:13.095309 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:13.095846  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:13.112214  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:13.196058  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:31:13.241569 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:13.303673 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 21:31:13.384843  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:13.387547 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:13.389607  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:13.398198 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:13.429153  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:13.431421  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:13.484305 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:13.657878  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 21:31:13.697317 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:13.698079  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:13.726255  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:13.737639  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:13.747509 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:13.776602 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:13.787475  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:13.792711 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:13.827230  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:31:13.937728  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:31:13.938755  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:31:13.953441  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:31:13.967988  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 21:31:14.052279  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:14.057477  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:14.066648 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:14.097908  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:14.111196 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:14.124922  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:14.177249 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:14.376611 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:14.401926  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:14.405642 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:14.405848 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:14.406519  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:14.413841  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:14.425920 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:14.429162  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:14.695951  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 21:31:14.731635 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:31:14.757197  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 21:31:14.758350  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:14.768178  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:14.781841  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:14.808902 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:14.838332  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:14.840695 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:14.881655 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:14.884984  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 21:31:15.100123  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:15.116632 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:15.121582  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:15.194228  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:15.214497 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:15.217000 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:15.221285  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:15.438612 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:15.468498  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:15.473510 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:15.479519  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:15.480011 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:15.488798  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:15.513712 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:15.518186  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 21:31:15.521219  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:15.695078  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:31:15.703701 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 21:31:15.821091  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:15.826778  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:15.836524 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:15.853075  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:15.853431 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:15.872280 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:15.875878  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:15.931833 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:16.210744 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:16.214888  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:16.224097 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:16.250234  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:16.259858 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:16.260411  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:16.270960  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:16.272055  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:31:16.323194  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 21:31:16.335460 11140 maintenance_manager.cc:419] P d10e9e156ba74a97a221c85b68ab8bc4: Scheduling FlushMRSOp(51936fb1f678458a8795f8246590addb): perf score=0.033347
W20240502 21:31:16.412689  9313 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.72:40467, user_credentials={real_user=slave}} blocked reactor thread for 63466.6us
W20240502 21:31:16.432754  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:31:16.490079 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:16.511976 11029 maintenance_manager.cc:643] P d10e9e156ba74a97a221c85b68ab8bc4: FlushMRSOp(51936fb1f678458a8795f8246590addb) complete. Timing: real 0.176s	user 0.089s	sys 0.063s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":85737,"dirs.queue_time_us":520,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":1075,"drs_written":1,"lbm_read_time_us":143,"lbm_reads_lt_1ms":4,"lbm_write_time_us":868,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":86202,"threads_started":3,"wal-append.queue_time_us":805}
I20240502 21:31:16.515748 11140 maintenance_manager.cc:419] P d10e9e156ba74a97a221c85b68ab8bc4: Scheduling UndoDeltaBlockGCOp(51936fb1f678458a8795f8246590addb): 4925 bytes on disk
I20240502 21:31:16.517444 11029 maintenance_manager.cc:643] P d10e9e156ba74a97a221c85b68ab8bc4: UndoDeltaBlockGCOp(51936fb1f678458a8795f8246590addb) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":188,"lbm_reads_lt_1ms":4}
W20240502 21:31:16.590130  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:31:16.590682  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 21:31:16.592561  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 21:31:16.663800  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:16.702021 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:16.707576  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:16.743470  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:16.771432 11140 maintenance_manager.cc:419] P d10e9e156ba74a97a221c85b68ab8bc4: Scheduling FlushMRSOp(07d693b680a54e12871121737e3da744): perf score=0.033370
I20240502 21:31:16.825193 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:16.827246 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:16.863055 11411 maintenance_manager.cc:419] P 4cbb5b06a6a44eb6b2145cd538b01af2: Scheduling FlushMRSOp(8b4486f03fd9474aa0bb2cb4c9149437): perf score=0.033378
I20240502 21:31:16.864668  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:16.929000 11029 maintenance_manager.cc:643] P d10e9e156ba74a97a221c85b68ab8bc4: FlushMRSOp(07d693b680a54e12871121737e3da744) complete. Timing: real 0.157s	user 0.051s	sys 0.010s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1031,"dirs.run_cpu_time_us":266,"dirs.run_wall_time_us":1897,"drs_written":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4,"lbm_write_time_us":904,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":17540,"threads_started":1,"wal-append.queue_time_us":17663}
I20240502 21:31:16.931151 11140 maintenance_manager.cc:419] P d10e9e156ba74a97a221c85b68ab8bc4: Scheduling UndoDeltaBlockGCOp(07d693b680a54e12871121737e3da744): 4584 bytes on disk
I20240502 21:31:16.931984 11029 maintenance_manager.cc:643] P d10e9e156ba74a97a221c85b68ab8bc4: UndoDeltaBlockGCOp(07d693b680a54e12871121737e3da744) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20240502 21:31:16.933425 11140 maintenance_manager.cc:419] P d10e9e156ba74a97a221c85b68ab8bc4: Scheduling FlushMRSOp(aeee59c5512d448d8f986f959dce1cc8): perf score=0.033397
I20240502 21:31:17.036571 11029 maintenance_manager.cc:643] P d10e9e156ba74a97a221c85b68ab8bc4: FlushMRSOp(aeee59c5512d448d8f986f959dce1cc8) complete. Timing: real 0.103s	user 0.038s	sys 0.014s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":159,"dirs.run_wall_time_us":1155,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":899,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":451,"threads_started":1,"wal-append.queue_time_us":583}
I20240502 21:31:17.038046 11140 maintenance_manager.cc:419] P d10e9e156ba74a97a221c85b68ab8bc4: Scheduling UndoDeltaBlockGCOp(aeee59c5512d448d8f986f959dce1cc8): 4586 bytes on disk
I20240502 21:31:17.038827 11029 maintenance_manager.cc:643] P d10e9e156ba74a97a221c85b68ab8bc4: UndoDeltaBlockGCOp(aeee59c5512d448d8f986f959dce1cc8) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
I20240502 21:31:17.065603 11300 maintenance_manager.cc:643] P 4cbb5b06a6a44eb6b2145cd538b01af2: FlushMRSOp(8b4486f03fd9474aa0bb2cb4c9149437) complete. Timing: real 0.201s	user 0.071s	sys 0.040s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":27632,"dirs.queue_time_us":6545,"dirs.run_cpu_time_us":262,"dirs.run_wall_time_us":4166,"drs_written":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1031,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":48537,"threads_started":3,"wal-append.queue_time_us":14403}
I20240502 21:31:17.068338 11411 maintenance_manager.cc:419] P 4cbb5b06a6a44eb6b2145cd538b01af2: Scheduling UndoDeltaBlockGCOp(8b4486f03fd9474aa0bb2cb4c9149437): 4586 bytes on disk
I20240502 21:31:17.081418 11300 maintenance_manager.cc:643] P 4cbb5b06a6a44eb6b2145cd538b01af2: UndoDeltaBlockGCOp(8b4486f03fd9474aa0bb2cb4c9149437) complete. Timing: real 0.013s	user 0.001s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":223,"lbm_reads_lt_1ms":4}
W20240502 21:31:17.092262 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:17.280112 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:31:17.323473  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 21:31:17.336988 11411 maintenance_manager.cc:419] P 4cbb5b06a6a44eb6b2145cd538b01af2: Scheduling FlushMRSOp(aeee59c5512d448d8f986f959dce1cc8): perf score=0.033374
I20240502 21:31:17.428916  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:17.446031  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:31:17.449182  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 21:31:17.478869 11300 maintenance_manager.cc:643] P 4cbb5b06a6a44eb6b2145cd538b01af2: FlushMRSOp(aeee59c5512d448d8f986f959dce1cc8) complete. Timing: real 0.141s	user 0.046s	sys 0.010s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":72,"dirs.run_cpu_time_us":239,"dirs.run_wall_time_us":2081,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":903,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":5071,"threads_started":1,"wal-append.queue_time_us":622}
I20240502 21:31:17.483233 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:17.484046 11411 maintenance_manager.cc:419] P 4cbb5b06a6a44eb6b2145cd538b01af2: Scheduling UndoDeltaBlockGCOp(aeee59c5512d448d8f986f959dce1cc8): 4586 bytes on disk
I20240502 21:31:17.484764 11300 maintenance_manager.cc:643] P 4cbb5b06a6a44eb6b2145cd538b01af2: UndoDeltaBlockGCOp(aeee59c5512d448d8f986f959dce1cc8) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":124,"lbm_reads_lt_1ms":4}
I20240502 21:31:17.486794 11411 maintenance_manager.cc:419] P 4cbb5b06a6a44eb6b2145cd538b01af2: Scheduling FlushMRSOp(cdf72d25f0f847e5998acbc53979e64d): perf score=0.033386
W20240502 21:31:17.490227 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:17.493554  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:31:17.494644  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 21:31:17.555423 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:17.558295  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:17.558534  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:17.566277 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:17.637650 11300 maintenance_manager.cc:643] P 4cbb5b06a6a44eb6b2145cd538b01af2: FlushMRSOp(cdf72d25f0f847e5998acbc53979e64d) complete. Timing: real 0.145s	user 0.050s	sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":10756,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":1670,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1010,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":8872,"threads_started":1,"wal-append.queue_time_us":8735}
I20240502 21:31:17.643011 11411 maintenance_manager.cc:419] P 4cbb5b06a6a44eb6b2145cd538b01af2: Scheduling FlushMRSOp(41ee481a3ece47a0905e93c66dbf949d): perf score=0.033394
W20240502 21:31:17.711582  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 21:31:17.783262 11544 maintenance_manager.cc:419] P e0de0545fa0547d3929111e990e6d1b2: Scheduling FlushMRSOp(a2f3edaf1c874cefb820f027f90d1185): perf score=0.033352
I20240502 21:31:17.843552 11300 maintenance_manager.cc:643] P 4cbb5b06a6a44eb6b2145cd538b01af2: FlushMRSOp(41ee481a3ece47a0905e93c66dbf949d) complete. Timing: real 0.193s	user 0.050s	sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":4212,"dirs.run_cpu_time_us":259,"dirs.run_wall_time_us":1683,"drs_written":1,"lbm_read_time_us":121,"lbm_reads_lt_1ms":4,"lbm_write_time_us":9298,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":7219,"threads_started":1,"wal-append.queue_time_us":7353}
I20240502 21:31:17.845546 11411 maintenance_manager.cc:419] P 4cbb5b06a6a44eb6b2145cd538b01af2: Scheduling UndoDeltaBlockGCOp(cdf72d25f0f847e5998acbc53979e64d): 4840 bytes on disk
I20240502 21:31:17.846318 11300 maintenance_manager.cc:643] P 4cbb5b06a6a44eb6b2145cd538b01af2: UndoDeltaBlockGCOp(cdf72d25f0f847e5998acbc53979e64d) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4}
I20240502 21:31:17.848265 11411 maintenance_manager.cc:419] P 4cbb5b06a6a44eb6b2145cd538b01af2: Scheduling UndoDeltaBlockGCOp(41ee481a3ece47a0905e93c66dbf949d): 4586 bytes on disk
I20240502 21:31:17.848950 11300 maintenance_manager.cc:643] P 4cbb5b06a6a44eb6b2145cd538b01af2: UndoDeltaBlockGCOp(41ee481a3ece47a0905e93c66dbf949d) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
W20240502 21:31:17.905958  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 21:31:17.971918 11433 maintenance_manager.cc:643] P e0de0545fa0547d3929111e990e6d1b2: FlushMRSOp(a2f3edaf1c874cefb820f027f90d1185) complete. Timing: real 0.188s	user 0.057s	sys 0.041s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":23838,"dirs.queue_time_us":15015,"dirs.run_cpu_time_us":301,"dirs.run_wall_time_us":7355,"drs_written":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1036,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":53027,"threads_started":3,"wal-append.queue_time_us":14858}
I20240502 21:31:17.989351 11544 maintenance_manager.cc:419] P e0de0545fa0547d3929111e990e6d1b2: Scheduling UndoDeltaBlockGCOp(a2f3edaf1c874cefb820f027f90d1185): 4841 bytes on disk
I20240502 21:31:17.991039 11433 maintenance_manager.cc:643] P e0de0545fa0547d3929111e990e6d1b2: UndoDeltaBlockGCOp(a2f3edaf1c874cefb820f027f90d1185) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":182,"lbm_reads_lt_1ms":4}
W20240502 21:31:18.056350 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:18.146971  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:18.183594  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:18.222607 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:18.241266  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:18.245186 11544 maintenance_manager.cc:419] P e0de0545fa0547d3929111e990e6d1b2: Scheduling FlushMRSOp(1497b781f36c43b39d22a05bb524c59d): perf score=0.033394
W20240502 21:31:18.280059 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 21:31:18.297114 11140 maintenance_manager.cc:419] P d10e9e156ba74a97a221c85b68ab8bc4: Scheduling FlushMRSOp(8b4486f03fd9474aa0bb2cb4c9149437): perf score=0.033355
W20240502 21:31:18.322904  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 21:31:18.341557 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:18.378594 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:18.391418  9173 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.69:42049, user_credentials={real_user=slave}} blocked reactor thread for 97659.2us
I20240502 21:31:18.481879  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:18.500936 11029 maintenance_manager.cc:643] P d10e9e156ba74a97a221c85b68ab8bc4: FlushMRSOp(8b4486f03fd9474aa0bb2cb4c9149437) complete. Timing: real 0.203s	user 0.051s	sys 0.040s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":45408,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":1674,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1012,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":9472,"thread_start_us":51993,"threads_started":2,"wal-append.queue_time_us":6840}
I20240502 21:31:18.503067 11140 maintenance_manager.cc:419] P d10e9e156ba74a97a221c85b68ab8bc4: Scheduling UndoDeltaBlockGCOp(8b4486f03fd9474aa0bb2cb4c9149437): 4586 bytes on disk
I20240502 21:31:18.503834 11029 maintenance_manager.cc:643] P d10e9e156ba74a97a221c85b68ab8bc4: UndoDeltaBlockGCOp(8b4486f03fd9474aa0bb2cb4c9149437) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
W20240502 21:31:18.512209  9039 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.69:42049, user_credentials={real_user=slave}} blocked reactor thread for 45928.8us
I20240502 21:31:18.564092 11433 maintenance_manager.cc:643] P e0de0545fa0547d3929111e990e6d1b2: FlushMRSOp(1497b781f36c43b39d22a05bb524c59d) complete. Timing: real 0.318s	user 0.069s	sys 0.031s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":39944,"dirs.run_cpu_time_us":305,"dirs.run_wall_time_us":10298,"drs_written":1,"lbm_read_time_us":154,"lbm_reads_lt_1ms":4,"lbm_write_time_us":980,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":61061,"threads_started":2,"wal-append.queue_time_us":18826}
I20240502 21:31:18.566541 11544 maintenance_manager.cc:419] P e0de0545fa0547d3929111e990e6d1b2: Scheduling UndoDeltaBlockGCOp(1497b781f36c43b39d22a05bb524c59d): 4840 bytes on disk
I20240502 21:31:18.567910 11433 maintenance_manager.cc:643] P e0de0545fa0547d3929111e990e6d1b2: UndoDeltaBlockGCOp(1497b781f36c43b39d22a05bb524c59d) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":160,"lbm_reads_lt_1ms":4}
I20240502 21:31:18.569514 11544 maintenance_manager.cc:419] P e0de0545fa0547d3929111e990e6d1b2: Scheduling FlushMRSOp(cffcb4f7f7754262beeb06168097f6dc): perf score=0.033467
W20240502 21:31:18.636476  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 21:31:18.652869 11433 maintenance_manager.cc:643] P e0de0545fa0547d3929111e990e6d1b2: FlushMRSOp(cffcb4f7f7754262beeb06168097f6dc) complete. Timing: real 0.083s	user 0.056s	sys 0.020s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":255,"dirs.run_wall_time_us":1936,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1013,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6779,"threads_started":1,"wal-append.queue_time_us":6589}
I20240502 21:31:18.654656 11544 maintenance_manager.cc:419] P e0de0545fa0547d3929111e990e6d1b2: Scheduling FlushMRSOp(a7c6b881c9ef4cdca787f065778e526f): perf score=0.033456
W20240502 21:31:18.657797 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:18.730696 11433 maintenance_manager.cc:643] P e0de0545fa0547d3929111e990e6d1b2: FlushMRSOp(a7c6b881c9ef4cdca787f065778e526f) complete. Timing: real 0.076s	user 0.062s	sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":1645,"drs_written":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1094,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":295,"threads_started":1,"wal-append.queue_time_us":478}
I20240502 21:31:18.732228 11544 maintenance_manager.cc:419] P e0de0545fa0547d3929111e990e6d1b2: Scheduling UndoDeltaBlockGCOp(a7c6b881c9ef4cdca787f065778e526f): 4925 bytes on disk
I20240502 21:31:18.732869 11433 maintenance_manager.cc:643] P e0de0545fa0547d3929111e990e6d1b2: UndoDeltaBlockGCOp(a7c6b881c9ef4cdca787f065778e526f) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240502 21:31:18.734048 11544 maintenance_manager.cc:419] P e0de0545fa0547d3929111e990e6d1b2: Scheduling UndoDeltaBlockGCOp(cffcb4f7f7754262beeb06168097f6dc): 4586 bytes on disk
I20240502 21:31:18.734582 11433 maintenance_manager.cc:643] P e0de0545fa0547d3929111e990e6d1b2: UndoDeltaBlockGCOp(cffcb4f7f7754262beeb06168097f6dc) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
W20240502 21:31:18.751372  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:31:18.755993  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 21:31:18.756412 11140 maintenance_manager.cc:419] P d10e9e156ba74a97a221c85b68ab8bc4: Scheduling FlushMRSOp(3caa36212ba444aa88d153ff9f90122e): perf score=0.033342
W20240502 21:31:18.808895  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 21:31:18.820176  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:18.859431 11411 maintenance_manager.cc:419] P 4cbb5b06a6a44eb6b2145cd538b01af2: Scheduling FlushMRSOp(87f35aff2b104c01a9d9d48a7e9c9b23): perf score=0.033342
I20240502 21:31:18.879946 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:31:18.880373  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 21:31:18.896359  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:18.915472  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:18.928830 11029 maintenance_manager.cc:643] P d10e9e156ba74a97a221c85b68ab8bc4: FlushMRSOp(3caa36212ba444aa88d153ff9f90122e) complete. Timing: real 0.172s	user 0.073s	sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1497,"drs_written":1,"lbm_read_time_us":121,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2827,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":46715,"threads_started":1,"wal-append.queue_time_us":46590}
I20240502 21:31:18.946238 11140 maintenance_manager.cc:419] P d10e9e156ba74a97a221c85b68ab8bc4: Scheduling UndoDeltaBlockGCOp(3caa36212ba444aa88d153ff9f90122e): 4923 bytes on disk
I20240502 21:31:18.947389 11029 maintenance_manager.cc:643] P d10e9e156ba74a97a221c85b68ab8bc4: UndoDeltaBlockGCOp(3caa36212ba444aa88d153ff9f90122e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":165,"lbm_reads_lt_1ms":4}
I20240502 21:31:18.947957  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:18.962924 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:18.994493 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:19.050849 11300 maintenance_manager.cc:643] P 4cbb5b06a6a44eb6b2145cd538b01af2: FlushMRSOp(87f35aff2b104c01a9d9d48a7e9c9b23) complete. Timing: real 0.191s	user 0.072s	sys 0.029s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":35427,"dirs.run_cpu_time_us":237,"dirs.run_wall_time_us":2258,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":898,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":49556,"threads_started":2,"wal-append.queue_time_us":14420}
I20240502 21:31:19.052558 11411 maintenance_manager.cc:419] P 4cbb5b06a6a44eb6b2145cd538b01af2: Scheduling UndoDeltaBlockGCOp(87f35aff2b104c01a9d9d48a7e9c9b23): 4840 bytes on disk
I20240502 21:31:19.054030 11300 maintenance_manager.cc:643] P 4cbb5b06a6a44eb6b2145cd538b01af2: UndoDeltaBlockGCOp(87f35aff2b104c01a9d9d48a7e9c9b23) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
W20240502 21:31:19.083223 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:19.201236 11140 maintenance_manager.cc:419] P d10e9e156ba74a97a221c85b68ab8bc4: Scheduling FlushMRSOp(8a2e06253c51405fa9b6bc9e87844760): perf score=0.033334
I20240502 21:31:19.208331  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:19.225668 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:19.242151  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:19.264122 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:19.270195  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:31:19.322166  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 21:31:19.335148  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:19.340857 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:19.409724 11029 maintenance_manager.cc:643] P d10e9e156ba74a97a221c85b68ab8bc4: FlushMRSOp(8a2e06253c51405fa9b6bc9e87844760) complete. Timing: real 0.207s	user 0.041s	sys 0.021s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":73,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":2835,"drs_written":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1716,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":5804,"threads_started":1,"wal-append.queue_time_us":5934}
I20240502 21:31:19.411520 11140 maintenance_manager.cc:419] P d10e9e156ba74a97a221c85b68ab8bc4: Scheduling UndoDeltaBlockGCOp(8a2e06253c51405fa9b6bc9e87844760): 4925 bytes on disk
I20240502 21:31:19.413558 11029 maintenance_manager.cc:643] P d10e9e156ba74a97a221c85b68ab8bc4: UndoDeltaBlockGCOp(8a2e06253c51405fa9b6bc9e87844760) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":687,"lbm_reads_lt_1ms":4}
W20240502 21:31:19.554296 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:19.586113  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:19.589004  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:19.593676 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:19.616209  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:19.623150 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:19.636631 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:19.642000  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:19.705296  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:31:19.765815  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 21:31:19.808861 11411 maintenance_manager.cc:419] P 4cbb5b06a6a44eb6b2145cd538b01af2: Scheduling FlushMRSOp(07d693b680a54e12871121737e3da744): perf score=0.033374
I20240502 21:31:19.888972 11300 maintenance_manager.cc:643] P 4cbb5b06a6a44eb6b2145cd538b01af2: FlushMRSOp(07d693b680a54e12871121737e3da744) complete. Timing: real 0.080s	user 0.054s	sys 0.022s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":7311,"dirs.run_cpu_time_us":277,"dirs.run_wall_time_us":3442,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":777,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":12074,"threads_started":2,"wal-append.queue_time_us":5132}
I20240502 21:31:19.891239 11411 maintenance_manager.cc:419] P 4cbb5b06a6a44eb6b2145cd538b01af2: Scheduling UndoDeltaBlockGCOp(07d693b680a54e12871121737e3da744): 4584 bytes on disk
I20240502 21:31:19.892220 11300 maintenance_manager.cc:643] P 4cbb5b06a6a44eb6b2145cd538b01af2: UndoDeltaBlockGCOp(07d693b680a54e12871121737e3da744) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":160,"lbm_reads_lt_1ms":4}
I20240502 21:31:19.894033 11411 maintenance_manager.cc:419] P 4cbb5b06a6a44eb6b2145cd538b01af2: Scheduling FlushMRSOp(8129d13163f64368b731ba4eaeaaa81c): perf score=0.033384
I20240502 21:31:19.917749 11140 maintenance_manager.cc:419] P d10e9e156ba74a97a221c85b68ab8bc4: Scheduling FlushMRSOp(de3287296c4547688388388dd4c7e638): perf score=0.033379
I20240502 21:31:19.983376  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:19.983227 11300 maintenance_manager.cc:643] P 4cbb5b06a6a44eb6b2145cd538b01af2: FlushMRSOp(8129d13163f64368b731ba4eaeaaa81c) complete. Timing: real 0.089s	user 0.048s	sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1615,"drs_written":1,"lbm_read_time_us":129,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1201,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":668,"threads_started":1,"wal-append.queue_time_us":521}
I20240502 21:31:19.985318 11411 maintenance_manager.cc:419] P 4cbb5b06a6a44eb6b2145cd538b01af2: Scheduling UndoDeltaBlockGCOp(8129d13163f64368b731ba4eaeaaa81c): 4584 bytes on disk
I20240502 21:31:19.986912 11300 maintenance_manager.cc:643] P 4cbb5b06a6a44eb6b2145cd538b01af2: UndoDeltaBlockGCOp(8129d13163f64368b731ba4eaeaaa81c) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":133,"lbm_reads_lt_1ms":4}
I20240502 21:31:19.988763 11411 maintenance_manager.cc:419] P 4cbb5b06a6a44eb6b2145cd538b01af2: Scheduling FlushMRSOp(a2f3edaf1c874cefb820f027f90d1185): perf score=0.033382
W20240502 21:31:19.998261 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:19.999459  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:20.002334 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:20.032531  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:20.046656 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:20.083936 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:20.087244  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:20.137240 11300 maintenance_manager.cc:643] P 4cbb5b06a6a44eb6b2145cd538b01af2: FlushMRSOp(a2f3edaf1c874cefb820f027f90d1185) complete. Timing: real 0.135s	user 0.052s	sys 0.027s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":716,"dirs.run_cpu_time_us":251,"dirs.run_wall_time_us":1584,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":843,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":29290,"threads_started":1,"wal-append.queue_time_us":29440}
I20240502 21:31:20.138939 11411 maintenance_manager.cc:419] P 4cbb5b06a6a44eb6b2145cd538b01af2: Scheduling UndoDeltaBlockGCOp(a2f3edaf1c874cefb820f027f90d1185): 4841 bytes on disk
I20240502 21:31:20.159754 11300 maintenance_manager.cc:643] P 4cbb5b06a6a44eb6b2145cd538b01af2: UndoDeltaBlockGCOp(a2f3edaf1c874cefb820f027f90d1185) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":124,"lbm_reads_lt_1ms":4}
W20240502 21:31:20.200073  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 21:31:20.240129 11029 maintenance_manager.cc:643] P d10e9e156ba74a97a221c85b68ab8bc4: FlushMRSOp(de3287296c4547688388388dd4c7e638) complete. Timing: real 0.322s	user 0.074s	sys 0.021s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":28391,"dirs.run_cpu_time_us":425,"dirs.run_wall_time_us":8876,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2545,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":28688,"threads_started":2,"wal-append.queue_time_us":424}
I20240502 21:31:20.241659 11140 maintenance_manager.cc:419] P d10e9e156ba74a97a221c85b68ab8bc4: Scheduling UndoDeltaBlockGCOp(de3287296c4547688388388dd4c7e638): 4926 bytes on disk
I20240502 21:31:20.242305 11029 maintenance_manager.cc:643] P d10e9e156ba74a97a221c85b68ab8bc4: UndoDeltaBlockGCOp(de3287296c4547688388388dd4c7e638) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
W20240502 21:31:20.365257  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 21:31:20.391681 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:20.395572  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:20.415472  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:20.427304  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:20.434746  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:20.449378 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:20.452140 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:20.551354 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:20.700999  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:20.716053 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:31:20.741061  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 21:31:20.750553  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:20.751591  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:20.777789 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:20.785769 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:20.811618 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 21:31:20.843595  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:20.922422  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:31:20.961643 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:20.997200 11140 maintenance_manager.cc:419] P d10e9e156ba74a97a221c85b68ab8bc4: Scheduling FlushMRSOp(be34ec89876d42b69efe1878c0d9bc19): perf score=0.033382
I20240502 21:31:20.997769 11544 maintenance_manager.cc:419] P e0de0545fa0547d3929111e990e6d1b2: Scheduling FlushMRSOp(f9ae8fac693d4d68b6741f2509e1265e): perf score=0.033395
I20240502 21:31:21.090638 11029 maintenance_manager.cc:643] P d10e9e156ba74a97a221c85b68ab8bc4: FlushMRSOp(be34ec89876d42b69efe1878c0d9bc19) complete. Timing: real 0.093s	user 0.052s	sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":13111,"dirs.run_cpu_time_us":386,"dirs.run_wall_time_us":2556,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1221,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":13568,"thread_start_us":19587,"threads_started":2,"wal-append.queue_time_us":6509}
I20240502 21:31:21.098616 11140 maintenance_manager.cc:419] P d10e9e156ba74a97a221c85b68ab8bc4: Scheduling UndoDeltaBlockGCOp(be34ec89876d42b69efe1878c0d9bc19): 4586 bytes on disk
I20240502 21:31:21.099880 11029 maintenance_manager.cc:643] P d10e9e156ba74a97a221c85b68ab8bc4: UndoDeltaBlockGCOp(be34ec89876d42b69efe1878c0d9bc19) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":166,"lbm_reads_lt_1ms":4}
I20240502 21:31:21.137282 11433 maintenance_manager.cc:643] P e0de0545fa0547d3929111e990e6d1b2: FlushMRSOp(f9ae8fac693d4d68b6741f2509e1265e) complete. Timing: real 0.139s	user 0.080s	sys 0.018s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":22143,"dirs.run_cpu_time_us":297,"dirs.run_wall_time_us":14134,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1111,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":34150,"threads_started":2,"wal-append.queue_time_us":11705}
I20240502 21:31:21.139586 11544 maintenance_manager.cc:419] P e0de0545fa0547d3929111e990e6d1b2: Scheduling UndoDeltaBlockGCOp(f9ae8fac693d4d68b6741f2509e1265e): 4838 bytes on disk
I20240502 21:31:21.144130 11433 maintenance_manager.cc:643] P e0de0545fa0547d3929111e990e6d1b2: UndoDeltaBlockGCOp(f9ae8fac693d4d68b6741f2509e1265e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":184,"lbm_reads_lt_1ms":4}
I20240502 21:31:21.158649  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:21.167526  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 21:31:21.169030 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:31:21.214087  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 21:31:21.222537  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:21.276118  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:21.276394 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:21.277052 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:21.291508  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 21:31:21.315699  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:21.417142 11411 maintenance_manager.cc:419] P 4cbb5b06a6a44eb6b2145cd538b01af2: Scheduling FlushMRSOp(f0259f01acb249dd99c662151dc21916): perf score=0.033339
W20240502 21:31:21.457067  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 21:31:21.479235 11300 maintenance_manager.cc:643] P 4cbb5b06a6a44eb6b2145cd538b01af2: FlushMRSOp(f0259f01acb249dd99c662151dc21916) complete. Timing: real 0.062s	user 0.044s	sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":9591,"dirs.run_cpu_time_us":289,"dirs.run_wall_time_us":2157,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1040,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":3328,"thread_start_us":13340,"threads_started":2,"wal-append.queue_time_us":3763}
I20240502 21:31:21.482481 11411 maintenance_manager.cc:419] P 4cbb5b06a6a44eb6b2145cd538b01af2: Scheduling UndoDeltaBlockGCOp(f0259f01acb249dd99c662151dc21916): 4840 bytes on disk
I20240502 21:31:21.483630 11300 maintenance_manager.cc:643] P 4cbb5b06a6a44eb6b2145cd538b01af2: UndoDeltaBlockGCOp(f0259f01acb249dd99c662151dc21916) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":127,"lbm_reads_lt_1ms":4}
I20240502 21:31:21.525418  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:21.525846  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:21.547276  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:21.550055 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:21.560160 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:21.565371 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:21.570859 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:21.578632  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:21.649320 11544 maintenance_manager.cc:419] P e0de0545fa0547d3929111e990e6d1b2: Scheduling FlushMRSOp(41ee481a3ece47a0905e93c66dbf949d): perf score=0.033395
W20240502 21:31:21.684592  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 21:31:21.735647 11433 maintenance_manager.cc:643] P e0de0545fa0547d3929111e990e6d1b2: FlushMRSOp(41ee481a3ece47a0905e93c66dbf949d) complete. Timing: real 0.086s	user 0.041s	sys 0.026s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":13970,"dirs.run_cpu_time_us":248,"dirs.run_wall_time_us":2609,"drs_written":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1466,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":16060,"threads_started":2,"wal-append.queue_time_us":2353}
I20240502 21:31:21.737126 11411 maintenance_manager.cc:419] P 4cbb5b06a6a44eb6b2145cd538b01af2: Scheduling FlushMRSOp(32c306d724954a16be21a34aa4cc0b5f): perf score=0.033349
I20240502 21:31:21.737423 11544 maintenance_manager.cc:419] P e0de0545fa0547d3929111e990e6d1b2: Scheduling FlushMRSOp(f82c35077224414cb284472d49d7de4f): perf score=0.033384
I20240502 21:31:21.800293 11433 maintenance_manager.cc:643] P e0de0545fa0547d3929111e990e6d1b2: FlushMRSOp(f82c35077224414cb284472d49d7de4f) complete. Timing: real 0.062s	user 0.039s	sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":53,"dirs.run_cpu_time_us":151,"dirs.run_wall_time_us":1348,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":940,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":4277,"threads_started":1,"wal-append.queue_time_us":4423}
I20240502 21:31:21.801146 11300 maintenance_manager.cc:643] P 4cbb5b06a6a44eb6b2145cd538b01af2: FlushMRSOp(32c306d724954a16be21a34aa4cc0b5f) complete. Timing: real 0.063s	user 0.035s	sys 0.022s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":71,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":2268,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1402,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":2925,"threads_started":1,"wal-append.queue_time_us":3049}
I20240502 21:31:21.801970 11544 maintenance_manager.cc:419] P e0de0545fa0547d3929111e990e6d1b2: Scheduling UndoDeltaBlockGCOp(41ee481a3ece47a0905e93c66dbf949d): 4586 bytes on disk
I20240502 21:31:21.802649 11411 maintenance_manager.cc:419] P 4cbb5b06a6a44eb6b2145cd538b01af2: Scheduling UndoDeltaBlockGCOp(32c306d724954a16be21a34aa4cc0b5f): 4840 bytes on disk
I20240502 21:31:21.802760 11433 maintenance_manager.cc:643] P e0de0545fa0547d3929111e990e6d1b2: UndoDeltaBlockGCOp(41ee481a3ece47a0905e93c66dbf949d) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4}
I20240502 21:31:21.804589 11544 maintenance_manager.cc:419] P e0de0545fa0547d3929111e990e6d1b2: Scheduling UndoDeltaBlockGCOp(f82c35077224414cb284472d49d7de4f): 4586 bytes on disk
I20240502 21:31:21.805430 11433 maintenance_manager.cc:643] P e0de0545fa0547d3929111e990e6d1b2: UndoDeltaBlockGCOp(f82c35077224414cb284472d49d7de4f) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240502 21:31:21.805442 11300 maintenance_manager.cc:643] P 4cbb5b06a6a44eb6b2145cd538b01af2: UndoDeltaBlockGCOp(32c306d724954a16be21a34aa4cc0b5f) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":136,"lbm_reads_lt_1ms":4}
I20240502 21:31:21.890002 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:21.909693  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:21.914628  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:21.938102 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:21.948681  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:21.951488 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:21.977834  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:22.106562 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:22.161486  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 21:31:22.281812  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:22.296483 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:22.297547 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:22.299626 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:22.299722  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:22.313074  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 21:31:22.320888  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:22.338028  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:22.357368 11140 maintenance_manager.cc:419] P d10e9e156ba74a97a221c85b68ab8bc4: Scheduling FlushMRSOp(f9ae8fac693d4d68b6741f2509e1265e): perf score=0.033334
W20240502 21:31:22.366420 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 21:31:22.482595 11029 maintenance_manager.cc:643] P d10e9e156ba74a97a221c85b68ab8bc4: FlushMRSOp(f9ae8fac693d4d68b6741f2509e1265e) complete. Timing: real 0.125s	user 0.061s	sys 0.021s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":11621,"dirs.run_cpu_time_us":371,"dirs.run_wall_time_us":15102,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":964,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":56905,"threads_started":2,"wal-append.queue_time_us":45574}
I20240502 21:31:22.488734 11140 maintenance_manager.cc:419] P d10e9e156ba74a97a221c85b68ab8bc4: Scheduling UndoDeltaBlockGCOp(f9ae8fac693d4d68b6741f2509e1265e): 4838 bytes on disk
I20240502 21:31:22.489596 11029 maintenance_manager.cc:643] P d10e9e156ba74a97a221c85b68ab8bc4: UndoDeltaBlockGCOp(f9ae8fac693d4d68b6741f2509e1265e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":128,"lbm_reads_lt_1ms":4}
W20240502 21:31:22.594235 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:22.647449  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:22.649101 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:22.660441 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:22.661594  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:22.664333  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:22.680444  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:22.684069 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:22.686026  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:31:22.700652  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:31:22.727085  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 21:31:22.903178  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:31:22.952785  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 21:31:23.027783  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:23.031775  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:23.033695 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:23.051402  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:23.065655 11544 maintenance_manager.cc:419] P e0de0545fa0547d3929111e990e6d1b2: Scheduling FlushMRSOp(79dedaa327b6479e8325f91481c1688e): perf score=0.033357
I20240502 21:31:23.074138  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:23.079627 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:23.115484 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:23.143028 28555 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:31:23.143276 28555 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:31:23.144199 28555 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:31:23.146955 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
W20240502 21:31:23.148315 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:23.148478 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:31:23.148815 28555 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 21:31:23.168388 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:23.268666 11433 maintenance_manager.cc:643] P e0de0545fa0547d3929111e990e6d1b2: FlushMRSOp(79dedaa327b6479e8325f91481c1688e) complete. Timing: real 0.203s	user 0.054s	sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":20760,"dirs.run_cpu_time_us":286,"dirs.run_wall_time_us":3014,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1137,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":27558,"threads_started":2,"wal-append.queue_time_us":6743}
I20240502 21:31:23.271728 11544 maintenance_manager.cc:419] P e0de0545fa0547d3929111e990e6d1b2: Scheduling UndoDeltaBlockGCOp(79dedaa327b6479e8325f91481c1688e): 4586 bytes on disk
I20240502 21:31:23.272707 11433 maintenance_manager.cc:643] P e0de0545fa0547d3929111e990e6d1b2: UndoDeltaBlockGCOp(79dedaa327b6479e8325f91481c1688e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
I20240502 21:31:23.461891 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:23.462389  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:23.469769 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:23.471462  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:23.485626  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:23.490890 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:23.492349  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:23.528084 11544 maintenance_manager.cc:419] P e0de0545fa0547d3929111e990e6d1b2: Scheduling FlushMRSOp(94d2cdf8ae6941668b5f680c558656e8): perf score=0.033353
W20240502 21:31:23.535449 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:31:23.559793  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 21:31:23.660602 11433 maintenance_manager.cc:643] P e0de0545fa0547d3929111e990e6d1b2: FlushMRSOp(94d2cdf8ae6941668b5f680c558656e8) complete. Timing: real 0.132s	user 0.076s	sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":73,"dirs.run_cpu_time_us":269,"dirs.run_wall_time_us":1549,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1109,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":10986,"threads_started":1,"wal-append.queue_time_us":11109}
I20240502 21:31:23.662603 11544 maintenance_manager.cc:419] P e0de0545fa0547d3929111e990e6d1b2: Scheduling UndoDeltaBlockGCOp(94d2cdf8ae6941668b5f680c558656e8): 4925 bytes on disk
I20240502 21:31:23.664213 11433 maintenance_manager.cc:643] P e0de0545fa0547d3929111e990e6d1b2: UndoDeltaBlockGCOp(94d2cdf8ae6941668b5f680c558656e8) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
W20240502 21:31:23.683068  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:31:23.694325  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:31:23.750917 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:23.793210  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 21:31:23.807076  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:23.811663 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:23.815171  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:23.819761  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:23.842042  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 21:31:23.848069 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:23.853016 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:23.857074  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:24.104107  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:24.122893 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:24.145704  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:24.149576  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:31:24.153930  9173 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 58606.7us
I20240502 21:31:24.161466 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:24.214356  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:24.215059 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:24.253458 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:24.256258 11140 maintenance_manager.cc:419] P d10e9e156ba74a97a221c85b68ab8bc4: Scheduling FlushMRSOp(e46065d94e874293a836bc74241b8e06): perf score=0.033364
I20240502 21:31:24.373011 11029 maintenance_manager.cc:643] P d10e9e156ba74a97a221c85b68ab8bc4: FlushMRSOp(e46065d94e874293a836bc74241b8e06) complete. Timing: real 0.116s	user 0.050s	sys 0.017s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":404,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":1831,"drs_written":1,"lbm_read_time_us":155,"lbm_reads_lt_1ms":4,"lbm_write_time_us":999,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":563,"threads_started":2,"wal-append.queue_time_us":542}
I20240502 21:31:24.375031 11140 maintenance_manager.cc:419] P d10e9e156ba74a97a221c85b68ab8bc4: Scheduling UndoDeltaBlockGCOp(e46065d94e874293a836bc74241b8e06): 4586 bytes on disk
I20240502 21:31:24.376194 11029 maintenance_manager.cc:643] P d10e9e156ba74a97a221c85b68ab8bc4: UndoDeltaBlockGCOp(e46065d94e874293a836bc74241b8e06) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":147,"lbm_reads_lt_1ms":4}
I20240502 21:31:24.487259  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:24.506484 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:24.507346  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:24.534159  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:24.539625 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:24.568701  9447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.72:40467, user_credentials={real_user=slave}} blocked reactor thread for 59895us
I20240502 21:31:24.582100 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:24.594349  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:24.597565  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:31:24.608857  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 21:31:24.843488 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:24.843976  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:31:24.849331 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:24.853582  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:24.856958 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:24.883813  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:24.890800 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:24.891139  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 21:31:24.908538  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:24.956854  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:31:25.016077  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 21:31:25.175581  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:25.207458 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:25.229743  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:25.243422  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:25.262362  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 21:31:25.279063 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:25.292922 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:25.313124  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:25.379273 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:25.579528  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:25.585140 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:25.585799  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:25.590679  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:25.611527 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:25.634093 11140 maintenance_manager.cc:419] P d10e9e156ba74a97a221c85b68ab8bc4: Scheduling FlushMRSOp(79dedaa327b6479e8325f91481c1688e): perf score=0.033355
I20240502 21:31:25.652374 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:25.721613  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:25.736470 11029 maintenance_manager.cc:643] P d10e9e156ba74a97a221c85b68ab8bc4: FlushMRSOp(79dedaa327b6479e8325f91481c1688e) complete. Timing: real 0.102s	user 0.077s	sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":689,"dirs.run_cpu_time_us":271,"dirs.run_wall_time_us":8691,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":881,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1173,"threads_started":2,"wal-append.queue_time_us":450}
I20240502 21:31:25.738068 11140 maintenance_manager.cc:419] P d10e9e156ba74a97a221c85b68ab8bc4: Scheduling UndoDeltaBlockGCOp(79dedaa327b6479e8325f91481c1688e): 4586 bytes on disk
I20240502 21:31:25.739854 11029 maintenance_manager.cc:643] P d10e9e156ba74a97a221c85b68ab8bc4: UndoDeltaBlockGCOp(79dedaa327b6479e8325f91481c1688e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
W20240502 21:31:25.856827 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:25.866483 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:31:25.920634  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 21:31:25.945569  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:25.946266 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:25.959287  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:25.983567  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:25.986129 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:26.003644  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:26.014742 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:26.071269  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:31:26.146324  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:31:26.237138  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:31:26.263443 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:26.308719  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 21:31:26.329404 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:26.335000  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:26.355094  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:26.383361  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:26.419598 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:26.466583 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:26.468400  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:26.687862  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:26.695802  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 21:31:26.702036  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:26.704022  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:26.704700 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:26.729249 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:26.730971 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:26.751173  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:26.791219 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:26.955026  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 21:31:27.030411  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:27.040194  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:27.046566 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:27.054803 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:27.068523  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:27.088274 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:27.088811  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:27.229364 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:27.308111  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 21:31:27.417115  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:27.420995  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:27.428829 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:27.437516 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 21:31:27.442289 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:27.443495  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:27.447021 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:27.456691  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:27.728660  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:27.739638 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:31:27.741497  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 21:31:27.765615  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:27.769156 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:27.774868  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:27.784415 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:27.818079  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:27.835216  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:31:27.837821 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:27.885447  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 21:31:27.886943  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 21:31:28.089455  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:28.097774 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:28.098392  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:28.127171 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:28.127816  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:28.144977  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:28.157016 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:28.163995  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:31:28.294898 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:31:28.416747 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:28.448791  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 21:31:28.464023  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:28.467747  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:28.469040 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:28.481025 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:28.483012  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:28.490396 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:28.496052  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:28.496690  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:31:28.636101  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:31:28.657543  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:31:28.773031  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 21:31:28.820053  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:28.828915  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:28.828965 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:28.836424  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:28.846807 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:28.849735 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:28.871608  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:28.944339 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:29.196950  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:29.209996 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:29.220259  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:29.226032  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:29.227088 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:29.243244  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:29.250442 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:29.415757  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:31:29.433017 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:29.521780 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:29.532303  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:29.548456  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:29.576277  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:29.576293 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:29.580740  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:29.687093 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:29.835896  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 21:31:29.900738  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:29.901458  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:29.904361 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:29.915253 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:29.915205  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:29.931187 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:31:29.933133 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:29.933656  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:30.023630  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:31:30.187964  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:31:30.227175  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 21:31:30.232055  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:30.233227 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:30.233897  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:30.240875 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:30.245715 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:30.246548  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:30.258052  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:31:30.378736 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:30.558624  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:30.578749 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:30.583629  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:30.583889  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:30.585220 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:30.587913 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:30.589195  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 21:31:30.609791  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:30.659679  9173 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 46165.7us
W20240502 21:31:30.770287  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:31:30.806573 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:31:30.873915  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 21:31:30.883286  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:30.888139  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 21:31:30.892720 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:31:30.904196 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:30.911928  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:30.917166  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:30.950605 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:30.973898 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:31.020215  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:31.150156  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 21:31:31.258706 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:31.259795  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:31.259963  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:31.279636 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:31.279891  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:31.293481 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:31.326014  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:31.446727 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:31.457098  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 21:31:31.552429  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:31.567960  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:31.577849  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:31.590931  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:31.602869 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:31.608887 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:31.610833 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:31.779978 28555 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:31:31.780230 28555 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:31:31.781009 28555 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:31:31.783632 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
W20240502 21:31:31.783895 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:31.784009 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:31:31.784373 28555 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 21:31:31.881014 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:31.901372 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:31.906810  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:31.911185  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:31.911772 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:31.922737 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:31.965546  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:32.026661  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:31:32.050304  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 21:31:32.148470  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 21:31:32.243058 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:32.243966 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:32.244220  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:32.272534  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:32.272876  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:32.275640  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:32.306700 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:31:32.393260 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:32.568234  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:32.580603  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:32.588807 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:32.601181  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:32.605332 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:32.621555 11411 maintenance_manager.cc:419] P 4cbb5b06a6a44eb6b2145cd538b01af2: Scheduling FlushMRSOp(1497b781f36c43b39d22a05bb524c59d): perf score=0.033395
W20240502 21:31:32.638083  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 21:31:32.690939  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:32.697765 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:32.759215  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 21:31:32.798987 11300 maintenance_manager.cc:643] P 4cbb5b06a6a44eb6b2145cd538b01af2: FlushMRSOp(1497b781f36c43b39d22a05bb524c59d) complete. Timing: real 0.177s	user 0.040s	sys 0.019s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":25998,"dirs.run_cpu_time_us":370,"dirs.run_wall_time_us":21843,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1018,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":26421,"threads_started":2,"wal-append.queue_time_us":376}
I20240502 21:31:32.800304 11411 maintenance_manager.cc:419] P 4cbb5b06a6a44eb6b2145cd538b01af2: Scheduling UndoDeltaBlockGCOp(1497b781f36c43b39d22a05bb524c59d): 4840 bytes on disk
I20240502 21:31:32.800998 11300 maintenance_manager.cc:643] P 4cbb5b06a6a44eb6b2145cd538b01af2: UndoDeltaBlockGCOp(1497b781f36c43b39d22a05bb524c59d) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
W20240502 21:31:32.819736 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 21:31:32.820858 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:32.920316  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 21:31:32.948678  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:32.949972  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:32.957862  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:32.957958 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:32.959007 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:32.967599 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:32.973726  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:33.008929  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 21:31:33.123579  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:31:33.215174 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:33.230085  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 21:31:33.277879 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:33.286075  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:33.287962  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:33.291536 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:33.318135 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:33.319561  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:33.351146 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 21:31:33.351467  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:33.399312  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:31:33.566617  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 21:31:33.610109 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:33.610785  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:33.615839  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:31:33.617127  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 21:31:33.619210 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:33.632134  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:33.647545 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:33.664678  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:33.754556 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:33.799693  9313 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.72:40467, user_credentials={real_user=slave}} blocked reactor thread for 77985.5us
W20240502 21:31:33.804118  9173 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.69:42049, user_credentials={real_user=slave}} blocked reactor thread for 114786us
W20240502 21:31:33.807078  9172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.71:35857, user_credentials={real_user=slave}} blocked reactor thread for 132649us
W20240502 21:31:33.810112  9310 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.71:35857, user_credentials={real_user=slave}} blocked reactor thread for 91112.3us
W20240502 21:31:33.837687  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:31:33.906854  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 21:31:34.039984  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:34.051900  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:34.062731 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:34.076798 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:34.086279 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:34.087177  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:34.096454 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:34.122399  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:34.387238  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:34.392659 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:34.416518  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:34.422221  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:34.450819 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:34.457351 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:34.466167  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:34.536540 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:34.610641  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 21:31:34.748136  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:34.751071  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:34.756729  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:34.776332 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:34.777791  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:34.786574 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:34.797912 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:31:34.911537 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:35.046087  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:35.060122 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:35.074404 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:35.076730  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:35.097684  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:35.112766 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:35.132782  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:35.149291  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:31:35.349179  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:31:35.349524  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:31:35.353346  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 21:31:35.412431 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:35.417002  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:35.430809  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:35.432452 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:35.440863  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:35.442618  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:35.443636 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:35.451659 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:35.473770  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:31:35.613564 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 21:31:35.724315  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:35.748781 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:35.755867  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:35.757339  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:35.760934  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:35.763192 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:35.778223 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:35.867133  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:31:36.037379  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 21:31:36.050848  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:36.061165  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:36.062850 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:36.079438 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:36.081395 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:36.092974  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:36.114305  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:36.115698 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:36.233827  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 21:31:36.407047  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:36.425689 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:36.431409  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:36.431731 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:36.450867 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:36.455626  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:36.456475  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:36.486510  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:31:36.574009  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:31:36.628738 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:36.709393  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:36.723683 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:36.738484  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:36.769845 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:36.807245 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:36.812425  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:36.892305  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:37.015770 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:37.054306  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 21:31:37.056046  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 21:31:37.146090 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:37.146926  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:37.169252  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:37.204128 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:37.207197 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:37.277083  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:37.277559  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:37.502589  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:37.505410 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:37.520166 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:37.527987  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:37.532836 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:37.533558  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:37.574175  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:37.582453  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:31:37.585333 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:37.702059  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 21:31:37.848980  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:37.855772  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:37.878245  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:37.896157 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:37.910634  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 21:31:37.912889 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:37.913228 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:37.923802  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:37.991470  9173 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 61627.8us
W20240502 21:31:38.041936 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 21:31:38.043208 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:31:38.105842 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:38.148882  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 21:31:38.208942 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:38.214046  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:38.215636  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:38.226876  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:38.244663  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:38.259850 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:38.273711 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:38.339282  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 21:31:38.344008  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:31:38.353547  9447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.72:40467, user_credentials={real_user=slave}} blocked reactor thread for 111739us
W20240502 21:31:38.407965  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:31:38.423360  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:31:38.482515 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:38.568405  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:38.579555 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:38.596863  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:38.601588  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:38.656134  9313 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.66:39071, user_credentials={real_user=slave}} blocked reactor thread for 73253.8us
I20240502 21:31:38.662397 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:38.705708 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:38.726903  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:38.793526  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:31:38.834925  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 21:31:38.978638  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:38.987957  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:38.991986  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:39.000393 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:31:39.018911 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:39.056406  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:39.058616 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:39.066205 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:39.163522  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 21:31:39.405097 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:39.406534  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:39.407938  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:31:39.422201  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 21:31:39.445605 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:39.450263  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:39.463413 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:39.474658  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:39.621479 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:39.754961  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:39.755901  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:39.760972  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 21:31:39.787319  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:39.787804 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:39.787926 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:39.806381  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:39.810164 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:40.093713  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:31:40.100028  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 21:31:40.114333  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:40.114974  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:40.129562 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:40.147583  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:40.150221 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:40.165128 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:40.166047  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:40.173856 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:40.358417  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 21:31:40.465440  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:40.472554 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:40.478654 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:40.478654  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:40.479173  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:40.497835  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:40.529170 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:40.581620  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:31:40.582926 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:31:40.615557 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:40.661983  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 21:31:40.791292  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:40.806740  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:40.808925  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:40.839416 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:40.878748 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:40.880511 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:40.881028  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:40.948336  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 21:31:41.108125  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:41.117739 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:41.122260  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:41.150050  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:41.167886 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:41.181970 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:41.206908  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:41.234869 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:41.355794  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 21:31:41.448397  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:41.449895  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 21:31:41.466073 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:41.468137  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:41.505651  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:41.514561 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:41.519491 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:41.532824  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:41.685802  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 21:31:41.784440  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:41.785372  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:41.785476 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:41.789786  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:31:41.801353 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:41.807724 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:41.816568 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:41.822824  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:41.906450  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:31:42.093829  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 21:31:42.116890  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:42.117867 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:42.120489  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:42.134683  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:42.178202 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:42.192368 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:42.243875 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:42.265753  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:42.361136  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 21:31:42.489346  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:42.501492  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:42.508075  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:42.516602 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:42.528864  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:42.535477 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:42.545841 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:42.672540  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:31:42.686841  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:31:42.775431 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:42.857859 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:42.863389  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:42.878041  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:42.886795 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:42.902035  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:42.903668  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:42.933113  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 21:31:42.938781 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:43.078478  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:31:43.143512  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 21:31:43.181340  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:43.204602 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:43.215901  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:31:43.234249 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:31:43.236428 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 21:31:43.240911  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:43.269909 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:43.273259 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:43.291819  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:43.373939 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:43.381549  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 21:31:43.536051  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:43.537231 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:43.539909  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:43.553764 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:43.562813  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:43.588778 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:43.607713  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:43.646188  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:31:43.718524  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 21:31:43.847523  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:43.867955 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:43.883143  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:43.883630  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:43.887564 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:43.919425  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:43.935753 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:43.973630 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:43.991245  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 21:31:44.152990  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:44.158680 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:44.163810  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:44.200130  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:44.205848 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:44.216434 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:44.227283  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 21:31:44.262084  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:44.321424  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:31:44.406792 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:44.565397  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:44.570894 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:44.578485  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:44.588261  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:44.615526 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:44.659819 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:44.663462  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:44.816322 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:44.913277  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:44.932313 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:44.943914  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:44.954413  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:44.959542 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:44.996719 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:44.998219  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:45.018675  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:31:45.090497  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:31:45.158293  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:31:45.193717  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 21:31:45.253588  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:45.255898 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:45.271351  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:45.281813 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:45.289978  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:45.299167 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:45.307883 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:45.339646  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:45.568511  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:45.571592 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:45.572289 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:45.573022  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:45.573340 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:45.576808  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:45.604125  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:31:45.630525  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:31:45.643674 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:31:45.688910  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 21:31:45.821297 11544 maintenance_manager.cc:419] P e0de0545fa0547d3929111e990e6d1b2: Scheduling FlushMRSOp(58c7004ceb044f5fa9dec54910046b69): perf score=0.033344
W20240502 21:31:45.832690  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:31:45.855837 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:45.875285 11433 maintenance_manager.cc:643] P e0de0545fa0547d3929111e990e6d1b2: FlushMRSOp(58c7004ceb044f5fa9dec54910046b69) complete. Timing: real 0.054s	user 0.038s	sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":8446,"dirs.run_cpu_time_us":265,"dirs.run_wall_time_us":2045,"drs_written":1,"lbm_read_time_us":128,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1633,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":13815,"threads_started":2,"wal-append.queue_time_us":5296}
I20240502 21:31:45.878283 11544 maintenance_manager.cc:419] P e0de0545fa0547d3929111e990e6d1b2: Scheduling UndoDeltaBlockGCOp(58c7004ceb044f5fa9dec54910046b69): 4925 bytes on disk
I20240502 21:31:45.879662 11433 maintenance_manager.cc:643] P e0de0545fa0547d3929111e990e6d1b2: UndoDeltaBlockGCOp(58c7004ceb044f5fa9dec54910046b69) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4}
I20240502 21:31:45.934564  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:45.955008 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:45.958597  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:45.962013 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:45.964812  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:45.975203 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:46.059763  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:46.289821 28555 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:31:46.290081 28555 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:31:46.291074 28555 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:31:46.292445 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
W20240502 21:31:46.293394 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:46.293561 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:31:46.293892 28555 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 21:31:46.314489 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:46.318231  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:46.343101  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:46.350772  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:46.362196 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:46.393574 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:46.394704 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:46.430178  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:46.446400  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:31:46.531271  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:31:46.607987  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 21:31:46.648381 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:46.648963  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:46.654516  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:46.672930 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:46.677946 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:46.680745  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:46.687420  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:46.865626 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:46.988313 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:46.988427  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:46.989578  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:46.996122 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:46.999904 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:47.008368  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:47.032002  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:47.160502  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:31:47.304452 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:47.317425  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:47.324957  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:47.324997  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:47.336702 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:47.340405 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:47.341519  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:47.350600 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:31:47.374018  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 21:31:47.399173  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:31:47.468259  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:31:47.503950  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 21:31:47.640502  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:47.648130 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:47.672472  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:47.689574  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:47.692116 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:47.701615  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:47.707882 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:47.754086 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:47.804337  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:31:47.853617  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 21:31:48.011889  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:48.028820 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:48.038761  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:48.051926 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:48.068271  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:48.069948  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:48.071813 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:48.274345  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 21:31:48.357242  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:48.363209 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:48.366427  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:48.410995 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:48.418426 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:31:48.441301  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 21:31:48.448042 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:48.452814  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:48.462374  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:48.467341 11430 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 48934.1us
W20240502 21:31:48.504266 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:31:48.504439 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:48.649330  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 21:31:48.730762  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:48.738384 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:31:48.758845  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 21:31:48.771909  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:48.793617 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:48.812459  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:48.824769 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:48.863914  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:48.885540 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:49.118736 11140 maintenance_manager.cc:419] P d10e9e156ba74a97a221c85b68ab8bc4: Scheduling FlushMRSOp(a7c6b881c9ef4cdca787f065778e526f): perf score=0.033336
I20240502 21:31:49.126937  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:49.144897 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:49.168933 11029 maintenance_manager.cc:643] P d10e9e156ba74a97a221c85b68ab8bc4: FlushMRSOp(a7c6b881c9ef4cdca787f065778e526f) complete. Timing: real 0.050s	user 0.040s	sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":583,"dirs.run_cpu_time_us":259,"dirs.run_wall_time_us":2452,"drs_written":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4,"lbm_write_time_us":980,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":12327,"threads_started":2,"wal-append.queue_time_us":11191}
I20240502 21:31:49.171548 11140 maintenance_manager.cc:419] P d10e9e156ba74a97a221c85b68ab8bc4: Scheduling UndoDeltaBlockGCOp(a7c6b881c9ef4cdca787f065778e526f): 4925 bytes on disk
I20240502 21:31:49.182463  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:31:49.185246 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:49.187140 11029 maintenance_manager.cc:643] P d10e9e156ba74a97a221c85b68ab8bc4: UndoDeltaBlockGCOp(a7c6b881c9ef4cdca787f065778e526f) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":172,"lbm_reads_lt_1ms":4}
W20240502 21:31:49.212332  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 21:31:49.212383  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:49.219722 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:49.234822 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:49.236939  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:49.358289  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:31:49.523245  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 21:31:49.536087  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:49.540344  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:49.548996 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:49.554168 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:49.564046  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:49.565618 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:49.585052  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:49.613535 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:49.856938  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:49.883790 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:49.891294  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:49.911064 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:49.926973  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:49.976434 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:49.988991  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 21:31:50.009050  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:50.039321  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:31:50.170651 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:50.174787  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:31:50.239831  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 21:31:50.242012  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:50.253743 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:50.271250  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:50.279755  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:50.282577  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:50.291978 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:50.336659 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:50.458781  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 21:31:50.560930  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:50.569737  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:50.571895 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:50.573129 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:50.586601 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:50.591763  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:50.595422  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:50.627187 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:50.663688 11544 maintenance_manager.cc:419] P e0de0545fa0547d3929111e990e6d1b2: Scheduling FlushMRSOp(51936fb1f678458a8795f8246590addb): perf score=0.033344
I20240502 21:31:50.741371 11433 maintenance_manager.cc:643] P e0de0545fa0547d3929111e990e6d1b2: FlushMRSOp(51936fb1f678458a8795f8246590addb) complete. Timing: real 0.077s	user 0.039s	sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":10230,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":891,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1002,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":11528,"threads_started":2,"wal-append.queue_time_us":348}
I20240502 21:31:50.743409 11544 maintenance_manager.cc:419] P e0de0545fa0547d3929111e990e6d1b2: Scheduling UndoDeltaBlockGCOp(51936fb1f678458a8795f8246590addb): 4925 bytes on disk
I20240502 21:31:50.744553 11433 maintenance_manager.cc:643] P e0de0545fa0547d3929111e990e6d1b2: UndoDeltaBlockGCOp(51936fb1f678458a8795f8246590addb) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4}
W20240502 21:31:50.768983 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:31:50.779513  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:31:50.872130  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 21:31:50.872567  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:50.886725  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:50.893818  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:50.907622 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:50.920110 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:50.921111  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:50.923086 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:31:51.102912 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:51.216809  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:51.236203 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:51.255792  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:51.268499 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:51.272202 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:51.322430  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:51.347630  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:31:51.362257  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:31:51.617157 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:51.630946  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:51.652287  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:51.658468 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:51.665169  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:31:51.667773  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 21:31:51.668943 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:51.695034 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:51.718374  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:51.751371 11544 maintenance_manager.cc:419] P e0de0545fa0547d3929111e990e6d1b2: Scheduling FlushMRSOp(4d1a31513bb64578a85adf1f622e5bc0): perf score=0.033340
W20240502 21:31:51.788033  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 21:31:51.839423 11433 maintenance_manager.cc:643] P e0de0545fa0547d3929111e990e6d1b2: FlushMRSOp(4d1a31513bb64578a85adf1f622e5bc0) complete. Timing: real 0.088s	user 0.041s	sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":400,"dirs.run_cpu_time_us":242,"dirs.run_wall_time_us":1829,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1061,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":9347,"threads_started":2,"wal-append.queue_time_us":1937}
I20240502 21:31:51.840965 11544 maintenance_manager.cc:419] P e0de0545fa0547d3929111e990e6d1b2: Scheduling UndoDeltaBlockGCOp(4d1a31513bb64578a85adf1f622e5bc0): 4840 bytes on disk
I20240502 21:31:51.841683 11433 maintenance_manager.cc:643] P e0de0545fa0547d3929111e990e6d1b2: UndoDeltaBlockGCOp(4d1a31513bb64578a85adf1f622e5bc0) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
I20240502 21:31:51.958290  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:51.974814 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:51.981756  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:51.987164 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:51.998466  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:52.012256  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:52.035928 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:52.111908 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:52.133467  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:31:52.263051  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 21:31:52.287103 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:52.291021  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:52.302846  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:52.306160 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:52.344702  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:52.353273 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:52.378252  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:52.409246  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 21:31:52.410641  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:31:52.496762  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:31:52.551931 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:52.617715  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:52.640434 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:52.644651 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:52.645793  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:52.647436  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:52.664119  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:52.665738 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:52.740577 32755 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:31:52.740792 32755 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:31:52.741504 32755 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:31:52.743139 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
W20240502 21:31:52.749557 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:52.749722 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:31:52.750124 32755 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 21:31:52.763712  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 21:31:52.952968  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:52.956187 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:52.966816 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:52.969026  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:52.969949 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:52.973423 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:52.986130  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:52.989663  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:53.215030  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 21:31:53.252513  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:53.258559 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:53.259233  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:53.281695  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:53.304641 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:53.316067 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:53.332628  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:53.360673 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:31:53.389075  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:31:53.432111 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240502 21:31:53.434304 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:53.527856  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240502 21:31:53.582017  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:53.604732 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:53.605026  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:53.605057 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:53.608335  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:53.623020 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:53.629127  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:53.866797  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:31:53.899730 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:53.934626  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:53.948992 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:53.955998  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:53.956425  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:53.962956 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:53.965183 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:53.976493  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:54.011485  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:31:54.070824  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:31:54.224889  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 21:31:54.258420  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:54.263695 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:54.268527  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:54.295831  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:54.303321 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:54.310745 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:54.339643  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:54.395234  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:31:54.491937 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:54.580703  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:54.581265 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:54.604861  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:54.630291  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:54.638504 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:54.654557 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:54.670151  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:54.696755  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:31:54.721532  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 21:31:54.938656  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:54.955430 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:54.972687  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:54.983425 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:54.985066 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:54.985618  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:55.000536  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:55.028604  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:31:55.106444 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:55.155474  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 21:31:55.286522  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:55.301256  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:55.317310 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:55.328804 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:55.329671  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:55.339336 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:55.359438  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:55.579885  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:55.596616  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:31:55.606422  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 21:31:55.607692 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:31:55.618701 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:55.625170  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:55.646080 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:55.649401 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:55.685377  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:55.758612 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:31:55.849087  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 21:31:55.960173  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:55.972283 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:55.973291  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:55.992311  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:55.999373  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:56.001392 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:56.016336 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:56.079550 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:56.263845  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 21:31:56.264901  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:56.267505  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:56.276295 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:56.293226 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:56.296911  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:56.316248  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:56.340121 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:56.430121  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:31:56.468441  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:31:56.533108 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:56.585876  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 21:31:56.615453  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:56.619544 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:56.623577  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:56.630244 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:56.631573  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:56.632107 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:56.649533  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:56.913623  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:56.914129 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:31:56.928669  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 21:31:56.929409 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:56.933156  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:56.938377  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 21:31:56.954893  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:56.956943  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:56.974620 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:57.038738 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:31:57.039315  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 21:31:57.250454  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:57.261678 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:57.263638  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:57.265453  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:57.269806 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:57.276245  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:57.306586 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:57.465906  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 21:31:57.525532  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:57.530274 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:31:57.541656 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:57.560055  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:57.560708 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:57.583621 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:57.584290  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:57.621155  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 21:31:57.624456  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:57.641347  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 21:31:57.886174  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:57.903692 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:57.917886 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:57.927778  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:57.928443  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:57.938719 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:57.951203  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:58.070340 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:31:58.075583  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:31:58.125095 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:58.253679  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:58.279858  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:58.284099 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:58.286167 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:58.309321  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:58.311430 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:58.322502  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:58.535732 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:31:58.555960  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 21:31:58.559430 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:58.651575  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:58.653767 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:58.667981  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:58.675961  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:58.691659  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:58.712519 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:58.713642 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:31:58.756435  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:31:59.005709  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:31:59.007252  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 21:31:59.020684  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:59.026135  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:31:59.035704 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:59.036887 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:31:59.050669  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 21:31:59.051156 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:59.054389 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:59.059221  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:59.059373  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 21:31:59.074254  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:59.140064  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:31:59.225683  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 21:31:59.344305  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:59.360451  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 21:31:59.363771 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:59.370152  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:59.385030 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:59.390074 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:59.395222  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:31:59.411759  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:31:59.583483 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:31:59.670993 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:31:59.672322  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:59.703919  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:31:59.744319 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:59.751327  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:31:59.774960 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:31:59.843364  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:31:59.905099  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 21:32:00.009794  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:00.010548 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:00.020956  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:00.036759 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:00.039203  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:00.040766 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:00.045073 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:00.068383  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:00.082520  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 21:32:00.329136  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:00.345647  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:00.357362  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:00.368247 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:00.379259 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:00.396211 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:00.421730 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:00.442288  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:00.460531  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:32:00.623129 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 21:32:00.696110 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:00.696134  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:00.710083  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:00.730038  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:00.733321 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:00.740707 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:00.787575  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:01.008245  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:32:01.029752 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:01.046514  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:01.054464 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:01.056543 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:01.064131  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:01.067849 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:01.068480  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:01.071170  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:32:01.245142  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 21:32:01.394464  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:01.394517 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:01.411793  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:32:01.428387  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 21:32:01.433670  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:01.434758  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 21:32:01.435179  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:01.436787 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:01.469079 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:01.494715  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:32:01.640566 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:01.652848  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:32:01.700745  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 21:32:01.722193 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:01.730055  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:01.730046  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:01.764379 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:01.775941  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:01.822760  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:01.826370 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:01.938426  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 21:32:02.033327  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:02.046649 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:02.062027 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:02.080791  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:02.084296  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:02.094938  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:02.099444 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:02.125387 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:02.386469  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 21:32:02.389402  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:02.397629 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:02.413615  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:02.420907  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:02.421358 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:02.434389 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:02.450147  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:02.524314 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:02.644632  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 21:32:02.659734  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 21:32:02.683861 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:02.685927  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:02.688513  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:02.688941 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:02.702504 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:02.718950  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:02.724182  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:02.833083  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 21:32:03.014340  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:03.020466  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:03.034406  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:03.036511 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:03.038563  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:03.040604 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:03.063283 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:32:03.116818 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:03.291152 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 21:32:03.353878  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:03.367604  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:03.375069 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:03.387526  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:03.392624 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:03.396920 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:03.425283  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:03.525080 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:03.577522  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 21:32:03.661619  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:03.666949 11544 maintenance_manager.cc:419] P e0de0545fa0547d3929111e990e6d1b2: Scheduling FlushMRSOp(4674e376bf7440cda619784491685b44): perf score=0.033391
I20240502 21:32:03.688735  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:03.689888  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:03.704196 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:03.739130 11295 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.68:40931, user_credentials={real_user=slave}} blocked reactor thread for 63709.5us
I20240502 21:32:03.744405 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:03.759629 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:03.764612 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 21:32:03.780895  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:03.842617  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:32:03.869764  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 21:32:03.890658 11433 maintenance_manager.cc:643] P e0de0545fa0547d3929111e990e6d1b2: FlushMRSOp(4674e376bf7440cda619784491685b44) complete. Timing: real 0.223s	user 0.047s	sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":19755,"dirs.run_cpu_time_us":253,"dirs.run_wall_time_us":3216,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":866,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":11250,"threads_started":2,"wal-append.queue_time_us":511}
I20240502 21:32:03.892729 11544 maintenance_manager.cc:419] P e0de0545fa0547d3929111e990e6d1b2: Scheduling UndoDeltaBlockGCOp(4674e376bf7440cda619784491685b44): 4925 bytes on disk
I20240502 21:32:03.893458 11433 maintenance_manager.cc:643] P e0de0545fa0547d3929111e990e6d1b2: UndoDeltaBlockGCOp(4674e376bf7440cda619784491685b44) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
W20240502 21:32:03.936121  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:32:04.045166 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:04.052701  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:04.064567 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:04.078002  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:04.078871  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:04.079654 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:04.093787 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:04.094870  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:04.243602  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:32:04.254563  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 21:32:04.281281  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:32:04.372515  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 21:32:04.394165  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:04.419405 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:04.429139 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:04.432966  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:04.456467  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:04.459715 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:04.472712  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:04.478636 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:04.567461  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 21:32:04.747758  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:04.762863 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:04.783542 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:04.786098  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:04.787796  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:04.813633 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:04.813956 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:04.834021  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:04.869138 32755 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:32:04.869410 32755 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:32:04.870476 32755 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:32:04.872756 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
W20240502 21:32:04.877853 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:04.878034 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:32:04.878335 32755 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 21:32:04.879488  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:32:04.982446  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:32:05.021549  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 21:32:05.174333  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:05.180971  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:05.188230  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:05.192370  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:05.209374 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:05.240926 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:05.271457 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:05.283291 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:05.501497  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:05.515340 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:05.521342  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:05.524566 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:05.524682  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:05.547791 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:05.568780  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:32:05.797519 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:05.813796 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:05.818614  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:05.819434  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:05.824859  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:05.840940  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:05.846930 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:05.848114 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:05.903805 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:32:06.021899  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 21:32:06.096364  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:06.117708  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:06.122915  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:06.132223 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:06.180758 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:06.199028  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:06.200683 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:06.371098 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:06.418902  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 21:32:06.473942  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:06.489434 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:32:06.494328  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 21:32:06.495834 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:06.495990  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:06.509562  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:06.545331 11140 maintenance_manager.cc:419] P d10e9e156ba74a97a221c85b68ab8bc4: Scheduling FlushMRSOp(4d1a31513bb64578a85adf1f622e5bc0): perf score=0.033369
I20240502 21:32:06.547520  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:06.555233 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:06.655543  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 21:32:06.679500 11029 maintenance_manager.cc:643] P d10e9e156ba74a97a221c85b68ab8bc4: FlushMRSOp(4d1a31513bb64578a85adf1f622e5bc0) complete. Timing: real 0.134s	user 0.028s	sys 0.035s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":7915,"dirs.run_cpu_time_us":286,"dirs.run_wall_time_us":2071,"drs_written":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1105,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":14714,"threads_started":2,"wal-append.queue_time_us":6700}
I20240502 21:32:06.682178 11140 maintenance_manager.cc:419] P d10e9e156ba74a97a221c85b68ab8bc4: Scheduling UndoDeltaBlockGCOp(4d1a31513bb64578a85adf1f622e5bc0): 4840 bytes on disk
I20240502 21:32:06.684173 11029 maintenance_manager.cc:643] P d10e9e156ba74a97a221c85b68ab8bc4: UndoDeltaBlockGCOp(4d1a31513bb64578a85adf1f622e5bc0) complete. Timing: real 0.002s	user 0.001s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":129,"lbm_reads_lt_1ms":4}
W20240502 21:32:06.701032  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:32:06.716958  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 21:32:06.841897  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:06.850224 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:06.868479 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:06.869223  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:06.869959  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:06.909548  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:06.932483  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 21:32:06.933641 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:06.982664 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:07.047713  9311 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.69:42049, user_credentials={real_user=slave}} blocked reactor thread for 54729.6us
I20240502 21:32:07.240309  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:07.269804  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:07.270951  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:32:07.277025  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:32:07.285883  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 21:32:07.291797 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:07.293058 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:07.324147  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:07.324647 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:07.458602 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:07.470889  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 21:32:07.637476  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:07.638900 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:07.643556  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:07.645784 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:07.648537  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:07.678737 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:07.679033  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 21:32:07.700485  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:07.781392  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 21:32:07.903100 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:07.973261  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:07.986801  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:07.989990 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:08.008904  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:08.014473  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:08.024852 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:08.047849 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:08.268282 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 21:32:08.326853  9290 maintenance_manager.cc:419] P 525a093ea5bf40dc9bbf9dc39895d670: Scheduling FlushMRSOp(f0259f01acb249dd99c662151dc21916): perf score=0.033341
I20240502 21:32:08.331815  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:08.341481 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:08.362993  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:08.373075 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:08.412127  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:08.420073 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:08.451056  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:08.480911 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:08.529335  9176 maintenance_manager.cc:643] P 525a093ea5bf40dc9bbf9dc39895d670: FlushMRSOp(f0259f01acb249dd99c662151dc21916) complete. Timing: real 0.202s	user 0.051s	sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":22146,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":1644,"drs_written":1,"lbm_read_time_us":135,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1023,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":29067,"threads_started":2,"wal-append.queue_time_us":664}
I20240502 21:32:08.530951  9290 maintenance_manager.cc:419] P 525a093ea5bf40dc9bbf9dc39895d670: Scheduling UndoDeltaBlockGCOp(f0259f01acb249dd99c662151dc21916): 4840 bytes on disk
I20240502 21:32:08.533066  9176 maintenance_manager.cc:643] P 525a093ea5bf40dc9bbf9dc39895d670: UndoDeltaBlockGCOp(f0259f01acb249dd99c662151dc21916) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":160,"lbm_reads_lt_1ms":4}
W20240502 21:32:08.664920  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240502 21:32:08.704107  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:08.704329  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:08.708673 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:08.716598  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:08.717450 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:08.725256  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:08.775218 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:08.819392 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:32:08.948597  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 21:32:09.005722  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:09.011746 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:09.015332  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:09.021171 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:09.027289  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:09.045558 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:09.046618  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:09.054005 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:09.142470  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:32:09.302614  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:32:09.320861  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:32:09.322340  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:32:09.337883  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 21:32:09.360075  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:09.361488  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:09.361642 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:09.377671 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:09.378913 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:09.392683  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:32:09.432358 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:09.433737  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:09.532372  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 21:32:09.669322  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:09.679749 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:09.697540 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:09.698647  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:09.703464 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:09.711701  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:09.719669  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:09.735085  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:32:09.792985  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:32:09.811972  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:32:09.977121 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:09.994261  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 21:32:10.022099  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:10.035275 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:10.035626  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:10.054284 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:10.070438  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:10.075094 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:10.095842  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:10.180889 11544 maintenance_manager.cc:419] P e0de0545fa0547d3929111e990e6d1b2: Scheduling FlushMRSOp(87f35aff2b104c01a9d9d48a7e9c9b23): perf score=0.033356
I20240502 21:32:10.302039 11433 maintenance_manager.cc:643] P e0de0545fa0547d3929111e990e6d1b2: FlushMRSOp(87f35aff2b104c01a9d9d48a7e9c9b23) complete. Timing: real 0.121s	user 0.070s	sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":7232,"dirs.run_cpu_time_us":230,"dirs.run_wall_time_us":2490,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1047,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":10366,"threads_started":2,"wal-append.queue_time_us":3232}
I20240502 21:32:10.304704 11544 maintenance_manager.cc:419] P e0de0545fa0547d3929111e990e6d1b2: Scheduling UndoDeltaBlockGCOp(87f35aff2b104c01a9d9d48a7e9c9b23): 4840 bytes on disk
I20240502 21:32:10.305598 11433 maintenance_manager.cc:643] P e0de0545fa0547d3929111e990e6d1b2: UndoDeltaBlockGCOp(87f35aff2b104c01a9d9d48a7e9c9b23) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4}
I20240502 21:32:10.439947  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:10.446229 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:10.446858  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:10.447605  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:10.460830  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:10.461966 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:10.479663 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:10.503427 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:10.619156 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 21:32:10.740614 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:10.755019  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:10.760870  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:10.786645  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:10.787135 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:10.805279 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:10.838642  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:11.047852 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:11.108628  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:11.119892  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:11.123821  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:11.124972 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:11.137091 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:11.140856 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:11.151978  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:11.190526  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:32:11.434041 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:11.457533  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:11.464856  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:11.466198  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:11.499332 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:11.510830 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:11.537087 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:11.542888  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 21:32:11.543970  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:11.595850  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:32:11.660248  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:32:11.748798  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 21:32:11.788708  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:11.796023 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:32:11.811340  8946 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:11.811432  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:32:11.813050  8946 catalog_manager.cc:4573] TS 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4d1a31513bb64578a85adf1f622e5bc0: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:11.815985  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:11.816865  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:11.852844 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:11.857105 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:11.860628  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:12.048008 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:12.087167  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 21:32:12.095538  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:12.101775  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:12.106014  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:12.109452 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:12.110366 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:12.112366 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:12.125227  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:12.128235  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:32:12.205324  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:32:12.313894  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 21:32:12.446404 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:12.447858  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:12.460036  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:12.475793  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:12.476451 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:12.496378  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:12.499354 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:12.516431 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:12.569375  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 21:32:12.758718  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:12.759927 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:12.765869  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:12.806613  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:32:12.816673  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 21:32:12.839059 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:12.851882 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:12.876775  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:13.097365 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:13.097486  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:13.099444 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:13.112962  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:13.120718  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:32:13.183550 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 21:32:13.200057 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:13.209790 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:13.225473  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:13.469089  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:13.482106  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:13.501395 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:13.507968  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:13.509653  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:13.520954 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:13.537617 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:13.627200 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:13.787515  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:32:13.790915  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 21:32:13.812294 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:13.813591 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:13.813834  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:13.822124  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:13.848269 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:13.866711  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:13.869657  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:32:13.934723  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 21:32:14.024765 11411 maintenance_manager.cc:419] P 4cbb5b06a6a44eb6b2145cd538b01af2: Scheduling FlushMRSOp(de3287296c4547688388388dd4c7e638): perf score=0.033374
W20240502 21:32:14.054216 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:32:14.055428 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:14.081962 11300 maintenance_manager.cc:643] P 4cbb5b06a6a44eb6b2145cd538b01af2: FlushMRSOp(de3287296c4547688388388dd4c7e638) complete. Timing: real 0.057s	user 0.041s	sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":1562,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":1180,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1212,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":29184,"thread_start_us":7644,"threads_started":2,"wal-append.queue_time_us":6661}
I20240502 21:32:14.083983 11411 maintenance_manager.cc:419] P 4cbb5b06a6a44eb6b2145cd538b01af2: Scheduling UndoDeltaBlockGCOp(de3287296c4547688388388dd4c7e638): 4926 bytes on disk
I20240502 21:32:14.084884 11300 maintenance_manager.cc:643] P 4cbb5b06a6a44eb6b2145cd538b01af2: UndoDeltaBlockGCOp(de3287296c4547688388388dd4c7e638) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4}
I20240502 21:32:14.149444  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:14.164877 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:14.167582  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:14.194983 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:14.196380  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 21:32:14.216552  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:14.230700 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:14.246654  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:14.332238  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:32:14.334995  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:32:14.439494  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 21:32:14.475037  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:14.478384 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:32:14.489679  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 21:32:14.499243 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:14.506414  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:14.506812  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:14.509168 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:14.515820  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:14.582235 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:14.774236  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 21:32:14.823921  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:14.828258  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:14.842494 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:14.846041 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:14.849267  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:32:14.855616  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:32:14.865080  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 21:32:14.877795  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:14.893509 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:14.906821  9446 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.67:43203, user_credentials={real_user=slave}} blocked reactor thread for 49696.9us
W20240502 21:32:14.942528  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:32:15.080462 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:15.169092  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:15.183894 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:15.198477  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:15.209592 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:15.229151  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:15.233587 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:15.301134  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:15.493379  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:15.503547  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:15.519263 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:15.530980 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:15.541373  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:15.575625  9040 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.67:43203, user_credentials={real_user=slave}} blocked reactor thread for 61837.8us
I20240502 21:32:15.579826  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:15.624017 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:15.702438 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:15.833681  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:15.836176 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:15.836198  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:15.839912  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:15.851197 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:15.853461 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:15.915860  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:15.937026 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:32:15.983620  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 21:32:16.123214  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:16.139880 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:16.163148  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:16.191854 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:16.204358 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:16.229002  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:16.237951  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:16.254876 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:16.317353  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 21:32:16.497706 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:16.497810  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:16.500190  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:16.512566 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:16.520316  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:16.537415  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:16.565436  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240502 21:32:16.566437 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:16.706539 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:16.735939  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:32:16.822338  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 21:32:16.848027  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:16.852691 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:16.868639 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:16.875934  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:16.877780  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:16.915400  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:16.937533 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:16.952706  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:32:17.023911  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:32:17.081851  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:32:17.114974 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:17.190994  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:17.200492 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:17.237303 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:17.243363  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:17.252589  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:17.270867 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:17.316231  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:17.486017  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:32:17.512888 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:17.535868  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 21:32:17.538487  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:17.538511  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:17.539734 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:17.542327  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:17.559728 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:17.594614 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:17.600883  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 21:32:17.621675  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:32:17.845963  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 21:32:17.854714 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:17.856734  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:17.881721  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:17.933749  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:17.951422  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:17.955040 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:18.000876 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:18.023813  9444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.71:35857, user_credentials={real_user=slave}} blocked reactor thread for 126991us
W20240502 21:32:18.140595 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:18.300050  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:18.325688  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:18.332664 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:18.336148 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:18.352197 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:18.355278  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:32:18.405359  9173 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 71744.1us
I20240502 21:32:18.412233  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:18.643668  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:18.646530 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:18.649533 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:18.655548  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:18.659233  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:18.662581 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:18.679741  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:18.711977 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:32:18.712327 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:18.743748  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:32:18.798415  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:32:18.967270  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 21:32:18.989059  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:19.001462  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:19.004179 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:19.027070  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:19.041167 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:19.066102 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:19.071766  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:19.082055  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:32:19.164845 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 21:32:19.168391 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:19.310181  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 21:32:19.416647  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:19.418021 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:19.430729  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:19.441552 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:19.441622  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:19.459421 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:19.486006  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:32:19.516975  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:32:19.517822  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:32:19.654326  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:32:19.725243  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 21:32:19.756062 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:19.756117  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:19.759840  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:32:19.776824 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:19.784786 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:19.788583 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:19.803087  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:19.827702  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:32:20.092123  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 21:32:20.107914  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:20.124564 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:20.130514  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:20.132977 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:20.152508  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 21:32:20.155154  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:32:20.189567 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:20.190480 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:20.205255  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:20.227171  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 21:32:20.457446  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:20.458125  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:20.462900 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:20.467381  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:20.475338 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:20.478827  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:20.518060 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:20.593758 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:20.754902  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:20.760129 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:20.769456  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:20.775584 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:20.781201 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:20.782094  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:20.819705  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:21.064357 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:21.071763  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:21.082471  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:32:21.100342 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:21.101599  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:21.103319 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:21.116832  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:21.125320  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 21:32:21.136565 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:21.284662 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:32:21.365341  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 21:32:21.441095  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:21.448489 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:21.450332  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:21.456219 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:21.457448  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:21.471391 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:21.473760  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:21.520251 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:21.545614  9447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.72:40467, user_credentials={real_user=slave}} blocked reactor thread for 46620.7us
W20240502 21:32:21.575009  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 21:32:21.800173  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:21.804570  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 21:32:21.805825  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:21.807300 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:21.817829  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:32:21.859707  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 21:32:21.908485  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:21.924714  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 21:32:21.933203 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:21.941509 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:21.953020 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:22.006094  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 21:32:22.012612  3488 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:32:22.012833  3488 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:32:22.013680  3488 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:32:22.015661 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
W20240502 21:32:22.015877 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:22.015980 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:32:22.016239  3488 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 21:32:22.113992  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:22.131670  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:22.145850 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:22.149258  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:22.150110 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:22.151733  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:22.207149 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:22.273856  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 21:32:22.426406  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:32:22.436355 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:22.453183 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:22.455053  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:22.462237 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:22.473703 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:22.498543  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:22.530462  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:22.580920  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:32:22.583978  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 21:32:22.819469  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:22.826371 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:22.835564  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:32:22.842018  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 21:32:22.846107  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:22.870729 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:22.877429  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 21:32:22.878073 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:22.878664 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:22.893898  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:23.189030  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:23.208719  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:23.209045 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:23.226898  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:23.241099 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:23.250530 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:23.267477  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:23.471859 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:23.529062 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:32:23.539053  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 21:32:23.539438  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:23.540725  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:23.545971 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:23.549891 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:23.551873  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:23.596644  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:23.694214  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:32:23.760838  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 21:32:23.869985  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:23.879981  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:23.879937 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:23.882841 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:23.884372  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:23.892951  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:23.894110 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:23.970853  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 21:32:24.005344 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:32:24.070178 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:24.156544  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:24.161664  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 21:32:24.162623 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:24.171810  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:24.202364  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:24.216316 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:24.231567 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:24.234340  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:24.250201  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:32:24.402212  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:32:24.489516 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 21:32:24.492159 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:24.531378  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:24.535212 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:24.536226  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:24.556133  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:24.568212 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:24.572254  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:24.599548 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:24.734253  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:32:24.818722  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:32:24.826809  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 21:32:24.845402  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:24.862675 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:24.874656  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:32:24.875710 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:24.898999 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:24.904387  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:24.920361 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:24.959651  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:24.988850  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 21:32:25.157486  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:25.164501 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:25.165447  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:25.170450  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:25.195581 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:25.257380  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 21:32:25.258002 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:25.295473  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:25.302721 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:25.570570  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:25.571252 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:25.586988  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:25.599651  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:25.615195 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:25.628325 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:25.659528  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:25.855621 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:25.927814  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:25.931677  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:25.940177 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:25.960911  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:25.964735  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:25.968498 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:25.971711 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:26.044628  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:32:26.199872  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 21:32:26.282300  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:26.291319 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:26.293051  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:26.299492  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:26.308233  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:26.309706 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:26.346729 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:26.386842  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:32:26.466603 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:26.491998 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:32:26.492439  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:32:26.537041  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 21:32:26.597160  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:26.614684 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:26.621214  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:32:26.637284  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 21:32:26.645440  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:26.665652 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:26.665827  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:26.684625 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:26.960780  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:32:26.992547 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:27.002048  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:27.008080  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:27.018141 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:27.018616  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:27.020324 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:27.023025  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 21:32:27.040818  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:27.041860 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:27.275184  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 21:32:27.325052  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:27.350031 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:27.351241  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:27.357888  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:27.381423 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:32:27.400724  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 21:32:27.402549 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:27.431499  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:27.466446 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:27.659073  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 21:32:27.698107  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:27.699748 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:27.709204  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:27.724869  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:27.739935 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:27.750735 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:27.760941  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:27.863094 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:28.017297  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:28.020623  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:28.025352  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:32:28.027493  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 21:32:28.028704 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:28.057559  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:28.058171 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:28.075759 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:28.284801  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:32:28.287708 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:28.309216  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:28.323089  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:28.334311 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:28.347513 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:28.357151  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:28.397671 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:28.419059  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:28.566308  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 21:32:28.704087  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:28.723007 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:28.733551  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:28.735319 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:28.750751  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:28.755156 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:28.757689 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:28.797555  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 21:32:28.799245  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:29.007397  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 21:32:29.059216 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:29.064332  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:32:29.070842  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 21:32:29.084950  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:29.109366 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:29.112859 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:29.115680  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:29.120016  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:29.152900  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:32:29.190337 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:29.219779  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:32:29.269526 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 21:32:29.422045  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:29.425427 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:29.430738  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:32:29.432466  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 21:32:29.447598  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:29.459656 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:29.480549 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:29.496465  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:29.638751  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:32:29.693984 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:32:29.699301 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:29.711968  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:32:29.798892  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 21:32:29.805665  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:29.807013 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:29.815305  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:29.842707 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:29.843549 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:29.844398  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:29.885152  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:30.143283  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:30.152499 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:30.167843  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:30.168598  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:30.175487  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:30.194499 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:30.199626 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:30.219201 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:30.410589  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 21:32:30.494639  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:30.496999 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:30.497396  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:30.502637 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:30.511643  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:30.526942  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:30.538029 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:30.667271  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:32:30.679904 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:30.842545  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:30.843710 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:30.853377  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:30.876806  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:30.908003 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:30.923897 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:30.965329  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:31.125211  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:32:31.231920  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 21:32:31.250458  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:31.258709 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:31.259215  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:31.269385 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:31.273828 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:31.276860  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:31.294266 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:31.327057  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:32:31.355433  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:32:31.492151  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 21:32:31.549063  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:31.568789 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:31.571211  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:31.585053  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:31.586071  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:31.588802 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:31.599071 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:31.682727  9173 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 58325.3us
W20240502 21:32:31.748437 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:31.821928  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:32:31.883440 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 21:32:31.954679 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:31.954792  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:31.967999  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:31.971428  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:31.977136 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:31.991823  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:31.991919 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:32.022331  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:32:32.166191  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:32:32.179971  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:32:32.223223  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 21:32:32.259814  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:32.274663  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:32.296070 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:32.296875 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:32.301980 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:32.304436  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:32.304616  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:32.359088 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:32.597235  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:32.600049  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:32.616525  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:32.617120 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:32.620525 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:32.621513 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:32.646039  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:32.743181  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:32:32.945801 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:32.988726 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:32.997629  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:33.003216  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:33.007759  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:33.018802 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:33.027051 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:33.042146  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:33.124279  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:32:33.212862  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240502 21:32:33.333552  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:33.348599  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:33.356150 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:33.362697 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:33.363690 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:33.364133  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:33.406140 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:33.432771  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:33.676575 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:33.682375  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:33.691612  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:33.706699 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:33.709861  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:33.729676 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:33.744818  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:32:33.747839  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 21:32:33.755997  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:33.869923 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:33.870281  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 21:32:34.001740  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:34.004602  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 21:32:34.017009  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:34.018621 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:34.021138  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:34.032485 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:34.034191  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:34.052390 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:32:34.142920  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 21:32:34.353597 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:32:34.358271 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:34.365823  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:34.367076  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:34.393810  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:32:34.423182 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 21:32:34.448124 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:34.454282  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 21:32:34.458195 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:34.505745  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:34.505687 11298 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.66:39071, user_credentials={real_user=slave}} blocked reactor thread for 53336us
W20240502 21:32:34.548786  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 21:32:34.739722  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:34.744197  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:34.748040 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:34.771835  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:34.800299 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:34.801414  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:34.808368 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:34.832790 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:32:34.860149  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:32:34.864629  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:32:34.923918 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:35.014775  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 21:32:35.080543  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:35.080621 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:35.084888  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:35.104833  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:35.109051 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:35.139757 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:35.150470  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:35.434844  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:35.442135 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:35.444123  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:35.447000 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:35.447506 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:35.459545  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 21:32:35.459986  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:35.490713  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:32:35.512631 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:35.762529 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:35.763373  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:35.771543  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:35.794543 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:35.795765 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:35.815048  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:35.816206  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:32:35.864020  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:32:36.014895 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:36.085896  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 21:32:36.096169  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:36.103082  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:36.103000  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:36.106665 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:36.137041 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:36.137547 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:36.170589  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:36.384606  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 21:32:36.434402  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:36.434574  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 21:32:36.441629 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:36.454471  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:36.459674 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:36.462937  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:36.471490 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:36.473955  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:36.556090  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:32:36.565634 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:36.733705  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:36.755497 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:36.774991  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:36.783421  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:36.822928 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:36.836846 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:36.843194  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:36.898329  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:32:37.015025 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:32:37.109185 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:37.133543  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:37.161989  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:37.165764 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:37.169914 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:37.170243  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:37.188522  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:37.219769  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 21:32:37.239333 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:37.292395  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 21:32:37.485055  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:37.485677 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:37.498809 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:37.514115  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:37.526573 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:37.527956  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:32:37.549728  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 21:32:37.575668  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:37.586230 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:37.721340  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:32:37.834386  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 21:32:37.851591  3488 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:32:37.851815  3488 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:32:37.852767  3488 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:32:37.860865 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
W20240502 21:32:37.863484 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:37.863689 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:32:37.864032  3488 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 21:32:37.880931  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:37.885288 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:37.900604  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:37.905530  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:37.925475 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:37.928160 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:37.930188  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:38.076545 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:38.205240  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:38.243395 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:32:38.261090  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 21:32:38.273090  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:38.280862 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:38.314950  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:38.315670 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:38.346580  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:38.403873  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 21:32:38.590631  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:38.592249 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:38.597783  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:38.624744 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:38.636502 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:38.638340  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:38.638424  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:38.664188 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:38.753772  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:32:38.779422  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 21:32:38.955904  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:32:38.964082  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 21:32:38.974241 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:32:38.977269  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 21:32:38.979983  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:38.981158  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:38.982781  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:38.989682 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:38.991425 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:39.046873 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:39.286866  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 21:32:39.295007  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:39.295648 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:39.301285 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:39.307798 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:39.309299  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:39.313395  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:39.320772  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:39.338111  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:32:39.422060 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:39.531666 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 21:32:39.643939  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:39.652815  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:39.670907  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:39.674757 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:39.701593 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:39.708307  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:39.710922 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:32:39.732383  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:32:39.892719 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:32:39.894281 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:39.946040  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:32:39.951148  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 21:32:39.988857  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:39.990994 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:39.995626  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:40.007213 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:40.018886 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:40.024343  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:40.058091  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:40.182343  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:32:40.272229  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:32:40.276669 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:40.314661  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:40.334445  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:40.339373 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:40.339970 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:40.352806  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:40.380249  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:40.409572 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:40.679085  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:40.680744 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:40.702971  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:40.706655 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:40.712996  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:40.750905  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:40.751739 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:40.820552  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:32:40.875357 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:41.001081  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:41.019227  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:41.020231 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:41.021898  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:41.028548 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:41.032362 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:41.105101  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:41.172158  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:32:41.188817  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:32:41.302012 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:41.318189  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:41.319314  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:41.321318 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:41.341395 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:41.350903  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:41.365593 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:41.399993  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 21:32:41.427347  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:41.665971  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 21:32:41.695533  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:41.698539  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:41.718888  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 21:32:41.724316 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:41.727964  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:41.732815 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:41.735888  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:41.783324 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:41.895788 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:42.044639 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 21:32:42.062397  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:42.067018  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:42.069310  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:42.086009 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:42.090579 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:42.109612  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:42.111402 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:42.240697  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:32:42.258069  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:32:42.378799 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:42.393270  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:42.410583 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:42.418279  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:42.422052  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:42.436558 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:42.447438 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:42.503800  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:42.601363  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:32:42.635754  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 21:32:42.734650  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:42.773449 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:32:42.778715  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 21:32:42.781024  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:42.802659  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:42.810463  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:42.828215 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:42.852663 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:42.903061 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:43.085258  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 21:32:43.110813  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:43.116941  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:43.124935 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:43.144697 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:43.149811 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:43.192520  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:43.218369  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:43.302367  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 21:32:43.365443 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:43.400265  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 21:32:43.522397 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:43.530097 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:43.544096  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:43.548657 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:43.572342  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:43.573635  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:43.596467  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:43.685235  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:32:43.794867 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:43.879638  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:43.901335 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:43.902256  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:43.918310 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:43.931347  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:43.940584 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:43.970512  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:44.003906  9311 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 95617.7us
W20240502 21:32:44.151262  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 21:32:44.252895 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:44.259603  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:44.271857  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:44.275287 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:44.291975  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:32:44.297331  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 21:32:44.298440  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:44.337518 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:44.366847 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:44.367610 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:32:44.525143  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:32:44.526454  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 21:32:44.572721 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:44.587721  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:44.594980  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:44.595494 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:44.621439  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:44.626029  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:44.661698 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:44.825546  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:32:44.860841  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:32:44.900450 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:32:44.903779 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:44.926041  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:44.949471 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:44.995857  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:44.997867 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:45.004280 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:45.018972  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:45.033522  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:32:45.146378  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:32:45.147678  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:32:45.206102  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 21:32:45.320667  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:45.333220  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:45.337774 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:45.347055  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:45.370992 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:45.395339  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 21:32:45.397589  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:45.401669 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:45.418011 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:45.641374  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:45.653903 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:45.659204  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:45.660174  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:45.669751 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:45.673792 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:45.709813  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:45.768926  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 21:32:45.975586  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:45.986959 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:46.001816  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:46.007246 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:46.007546 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:46.028334  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:32:46.042076 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:46.062831  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 21:32:46.063558  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:46.365892 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:46.366061  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:46.381117  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:46.407039  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:46.416849 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:46.426769 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:46.464102 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:46.466506  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:46.557046 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:32:46.648093  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:32:46.670471  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 21:32:46.722916 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:46.724363  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:46.726589  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:46.757618  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:46.766088 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:46.826736 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:46.836061 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:46.839773  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:47.054236  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 21:32:47.112011 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:47.112769  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:47.113698 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:47.126785 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:47.147015  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:47.167856  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:47.216306  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:47.378079  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:32:47.435582 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:47.436105  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:47.452009  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:47.455392  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:47.457062 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:47.461140 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:47.503918 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:47.543443  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:47.544337  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:32:47.821772  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 21:32:47.832470  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:47.841646 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:47.852428  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:47.865852 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:47.866655  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:47.873493 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:47.880504  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:32:47.881606  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:32:47.893054  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 21:32:47.893071 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:47.943858  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:48.055565  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 21:32:48.149195 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:48.150017  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:48.168112  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:48.173413  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:48.186444  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:48.187474 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:48.215224 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:48.344547 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:48.436182  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 21:32:48.470338  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:48.484107  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:48.486734  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:48.488137 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:48.497912  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:48.511751 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:48.519171 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:48.768893  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 21:32:48.794178  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:48.808275  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:48.821434 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:48.838487 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:48.839082  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:48.843672  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:48.851600 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:48.868373 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:48.965492 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:32:49.022534  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 21:32:49.145035  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:49.145114  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:49.171281  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:49.174789 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:49.183183 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:49.215502  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:49.243239 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:49.334547 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:49.433316  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:32:49.434264  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 21:32:49.553046  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:49.558303  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:49.560755 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:49.561147  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:49.561769 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:49.597318 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:49.603451  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 21:32:49.606436  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:49.795693 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:49.910418  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:49.911243  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:49.941270 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:49.946457 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:49.954530  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:49.970460  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:49.973160  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 21:32:49.973888 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:50.036000  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:32:50.180913  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:32:50.268618  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 21:32:50.294168  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:32:50.295305  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:32:50.297070  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 21:32:50.312036 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:50.330610 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:50.331502 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:50.332785  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:50.357244  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:50.394660  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:50.399427  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:32:50.417559 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 21:32:50.437158 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:50.628803  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:50.629367 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:50.643574  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:50.651489  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:50.664961 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:50.665089 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:50.683645  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:50.829506 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:50.962116  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:50.971657 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:50.979974  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:50.982299 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:50.988433  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:51.005407 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:51.052527  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:51.118949  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 21:32:51.298547  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:51.311565 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:51.314878  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:51.332144 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:51.353515  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:51.364699 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:51.410620  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:51.440065 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:32:51.444064 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:51.548804  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 21:32:51.636814  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:51.654237 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:51.657722  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:51.670049  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:51.674218 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:51.687453 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:51.717077  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:51.800338  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:32:51.859800  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:32:51.982501 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:52.021898 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:52.045789  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:52.048677  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:52.051394 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:52.052788  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:52.068107 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:52.081686  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 21:32:52.090415  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:32:52.356261  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:32:52.360968  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 21:32:52.407209  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:52.411863  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:52.420221 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:52.420732  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:52.425572 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:52.436098 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:52.458434  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:52.516673 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:52.597380  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:32:52.688728  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 21:32:52.721437 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:52.721915  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:52.733739  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:52.752362 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:52.752954  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:52.767694 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:52.784073  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:32:52.893997  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:32:52.993074  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:32:53.014178 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:53.078169 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:53.078948  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:53.094759 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:53.103438  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:53.111853  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:53.114372  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:53.146312 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:53.429260  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:53.444973  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:53.446514 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:53.447747 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:53.453068  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:53.463493 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:53.475399  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 21:32:53.504654  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:53.602722 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:53.697599  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:32:53.721487 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 21:32:53.735044 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:53.739768  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:53.747740  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:53.749044  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:53.839321 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:53.854038 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:53.856585  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:53.974115  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 21:32:54.101239  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:54.120404  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:54.124109  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:54.127560  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:54.151666 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:54.152220 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:54.172263 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:54.172377 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:54.437390  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:54.446264 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:32:54.461364  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 21:32:54.463047  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:54.468384 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:54.489039  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:54.490345 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:54.531313  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:54.600142  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:32:54.630200 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:54.670986  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:32:54.733067  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 21:32:54.762413  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 21:32:54.846486  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:54.847146  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:54.874626  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:54.888109 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:54.905004 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:54.941584  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:54.965760 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:54.983395  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:32:55.006410  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 21:32:55.222896  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:55.223079 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:55.258924  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:55.264498  7667 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:32:55.264755  7667 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:32:55.265718  7667 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:32:55.271986 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
W20240502 21:32:55.272223 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:55.272322 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:32:55.272629  7667 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 21:32:55.273849 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:55.280148 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:55.295799  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:55.306911  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:55.321898 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:32:55.333590  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:32:55.510782  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:32:55.512168  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 21:32:55.608026 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:55.618868  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:55.626122  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:55.643445  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:55.689800 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:55.692728 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:55.694110 11028 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.72:40467, user_credentials={real_user=slave}} blocked reactor thread for 43586.8us
W20240502 21:32:55.714869 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 21:32:55.716605 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:55.732090  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:55.949019  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:55.982888 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:55.983965  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:55.986122  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:56.013943 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:56.024907 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:56.062278  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:56.223657 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:56.236371  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 21:32:56.328364  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:56.337920 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:56.342381  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:32:56.359155 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 21:32:56.373347 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:56.382128  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:56.387465 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:56.388263  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 21:32:56.427248  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:56.711921  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:56.715085  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:56.727841 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:56.729470 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:32:56.747081 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:56.771940 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:56.774572  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:56.808874  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:56.906153  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:32:57.012666  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 21:32:57.034185 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:57.034652  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:57.063964  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:57.064147  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:57.076947  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:57.080531 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:57.095465 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:57.238946  7667 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:32:57.239495  7667 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:32:57.240612  7667 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:32:57.242470 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
W20240502 21:32:57.242661 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:57.242775 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:32:57.243120  7667 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 21:32:57.253666  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:32:57.289858  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:32:57.290526  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:32:57.307782 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:57.426894  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 21:32:57.438373  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:57.442602 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:57.445629  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:57.456326  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:57.483595 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:57.493940 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:57.528306  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:57.724732  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 21:32:57.762229  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:57.773018 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:57.775322  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:57.792830 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:57.821148  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:57.857095 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:57.864907  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:57.898221 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:58.058128  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 21:32:58.119833  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:58.129405  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:58.132059 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:58.136557 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:58.149116  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:58.150907  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:58.154561  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 21:32:58.158910 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:58.460175  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:58.461716  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:58.474972 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:58.482595  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:32:58.483307 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:32:58.483366  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 21:32:58.486132 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:58.487424  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:58.515044 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:58.785993 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:58.788105  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:58.798398  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:58.818876 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:58.827086 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 21:32:58.840052  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:32:58.893896  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 21:32:58.913311 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:58.954239 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:58.960386  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:59.104677  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 21:32:59.135063 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:59.142251  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:59.147818  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:59.158763  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:59.191388 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:59.260957 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:59.274046  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:32:59.379580  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 21:32:59.480428  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:59.501127 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:59.506583  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:59.508682  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:32:59.509759 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:32:59.514413 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:32:59.525180  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:59.555843 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:59.589578  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:32:59.643718  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:32:59.844195  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 21:32:59.847383  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:32:59.868121  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:32:59.868594  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:59.882700 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:32:59.884230  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 21:32:59.886132 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:32:59.897541  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:32:59.900347 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:00.031459  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:33:00.052538 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:00.100379  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 21:33:00.207077  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:00.232607 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:00.246784  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:00.275883  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:00.318679  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 21:33:00.334671 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:00.362442 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:00.363777  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:00.553457  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:33:00.555516  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 21:33:00.572436  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:00.583052  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:00.596508 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:00.614864  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:00.618695 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:00.632889 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:00.679425 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:00.701306  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:00.852339 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 21:33:00.973954  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:00.995054 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:01.012348  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:01.017740 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:01.023844  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:01.025028 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:01.056502  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:01.182879 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:01.302466  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:01.307549  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:01.311614 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:01.316442  7667 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:33:01.316640  7667 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:33:01.317353  7667 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:33:01.319062 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:01.322504 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
I20240502 21:33:01.322647 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:33:01.323158  7667 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 21:33:01.326021 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:01.343744  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:01.359807  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:01.368500 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:01.399230 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:33:01.487692  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:33:01.596546  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 21:33:01.654779  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:01.659010  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:01.661528 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:01.665678  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:01.666064  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:01.673334 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:01.697142 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:01.703979  9172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.71:35857, user_credentials={real_user=slave}} blocked reactor thread for 49996.8us
W20240502 21:33:01.707201 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:01.853879  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 21:33:02.010378  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:02.025358  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:02.030417 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:02.042316 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:02.042833  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:02.042966 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:02.062827  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:02.083201  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:33:02.120795  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:33:02.267038 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:02.306097  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:33:02.319928  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 21:33:02.350509  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:02.356487  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:02.375248 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:02.376289  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:02.394536  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:02.400048 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:02.406266 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:02.621088  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 21:33:02.642966  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:02.643458  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:02.664714  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:02.670055 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:02.671401 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:02.673717 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:02.693660  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:33:02.769079 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:02.789518  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:02.997898 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:03.000530  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:03.001153  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:03.024863 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:03.040095 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:03.041035  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:03.073259  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:03.232757  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:33:03.242856 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:03.299365  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:33:03.313416  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 21:33:03.318305 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:03.319012 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:03.319793  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:03.322126  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:03.349576  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:03.354151 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:03.545496  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 21:33:03.631494  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:03.631604 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:03.634140  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:03.656306 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:03.656843  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:03.664216 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 21:33:03.669473 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:03.676376  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:03.720633 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:03.986487  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:33:03.995517  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 21:33:03.999879  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:04.006929 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:04.010073  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:04.010555  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:04.011909 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:04.022873 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:33:04.064541  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:33:04.105747 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:04.303519  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 21:33:04.356866  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:04.357421  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:04.360620  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:04.379789 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:04.385025 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:04.408720  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:04.444550 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:04.462669  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:33:04.574267 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:04.593778  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 21:33:04.681354 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:04.690052  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:04.697239  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:04.730391 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:04.731437  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:04.731940 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:04.742928  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 21:33:04.768591  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:04.868407  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:33:04.914714  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 21:33:05.079031 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:05.079762  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:05.116990  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:05.125295 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:05.126543  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:05.136376  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:05.158686 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:05.198910 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:05.209956  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 21:33:05.445508  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:05.456404 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:05.463744  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:05.473594  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:05.477762  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 21:33:05.493805 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:05.507414 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:05.572535  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:05.606739 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:05.789763  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:05.795181 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:05.798712  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:33:05.817322  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:33:05.818318  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 21:33:05.820936 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:05.828481  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:05.841559 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:05.855669  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:06.005995 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 21:33:06.007225 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 21:33:06.157498  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:06.178290  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:06.181676  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:06.185009 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:06.204697  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:06.220531 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:06.267726 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:06.269086 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:06.415983  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 21:33:06.548213  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:06.563678  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:06.576818 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:06.582108 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:06.584268  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:06.586580 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:06.599272  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:06.654618  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:33:06.829437  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:33:06.830134 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:06.869225  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 21:33:06.946573  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:06.951856 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:06.962342 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:06.976472  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 21:33:06.992019  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:07.019954  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:07.028331  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:07.057430 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:07.238641  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 21:33:07.321960  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:07.327803  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 21:33:07.331699 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:07.339861  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:07.350203  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:07.370435 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:07.394697 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:07.403215 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:07.434270  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:07.460712  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 21:33:07.652366  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:07.660291 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:07.666335  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:07.667841 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:07.687260 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:07.691448  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:07.732261  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:07.870910 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:07.939157  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 21:33:08.006173  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:08.021198 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:08.031908 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:08.040431  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:08.051026  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:08.052635  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:08.054096 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:08.300321 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:08.346694  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:08.355823 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:08.360792  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:08.370024  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:08.401297  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 21:33:08.414053 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:08.417649 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:08.419478  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:08.607447  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 21:33:08.622589  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:33:08.667258 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 21:33:08.681262  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:08.691715  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:08.692265  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 21:33:08.694481 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:08.709205 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:08.710844  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:08.712630 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:08.721056 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:08.724491  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:08.808569  9038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.71:35857, user_credentials={real_user=slave}} blocked reactor thread for 50120.9us
I20240502 21:33:08.993450  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:09.009506  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:09.016883 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:09.025063 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:09.028728  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 21:33:09.029480  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:09.048362 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:09.077433  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:09.086386  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:33:09.107679 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:09.198529  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 21:33:09.349437  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:09.370580  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:09.391839  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:09.417548 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:09.420886 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:09.422727 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:09.457505  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:09.467316  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:33:09.639434 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:09.689471  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:09.714541 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:09.715456  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:09.733879 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:09.750990  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:09.753127 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:09.804651  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:09.884646  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 21:33:09.900424  9993 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:33:09.900660  9993 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:33:09.901470  9993 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:33:09.903461 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
W20240502 21:33:09.904374 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:09.904541 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:33:09.907650  9993 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 21:33:09.937608  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 21:33:10.025106  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:33:10.058003 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:10.064705  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:10.079164  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:33:10.105453  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 21:33:10.107630 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:10.125406 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:10.152521  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:10.153334  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:10.162669 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:10.211272  9173 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 55591.7us
I20240502 21:33:10.494024 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:10.494457  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:10.514833  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:10.515915 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:10.522254  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:10.528646 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:10.538241  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 21:33:10.549196  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:10.584285 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:10.816354  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:10.821631 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:10.837327  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:10.850739  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:33:10.866235  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:33:10.867621  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 21:33:10.873610  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:10.874809 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:10.885702 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:10.927491 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:11.118537 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 21:33:11.119837 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:33:11.154662  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 21:33:11.181788  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:11.193972  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:11.196120 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:11.214344  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:11.239328 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:11.246381 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:11.277626  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:11.412746 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:11.531024  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:33:11.553999  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 21:33:11.582110 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:11.585470  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:11.595572  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:11.598690  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:11.633734 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:11.645453 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:11.655968  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:11.827961  8946 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:11.829136  8946 catalog_manager.cc:4573] TS 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4d1a31513bb64578a85adf1f622e5bc0: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:11.913873  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:11.914865 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:11.916791  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:11.941910 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:11.946044  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:11.952281 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:11.960556  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 21:33:11.965492 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:11.968559  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:12.191439  9993 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:33:12.191722  9993 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:33:12.192849  9993 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:33:12.195423 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
W20240502 21:33:12.197335 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:12.197674 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:33:12.198554  9993 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 21:33:12.201085  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 21:33:12.247704  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:12.266454 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:12.266904 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:12.278685  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:12.281340  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:12.292925  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:12.303511 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:12.323262  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:33:12.429224  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:33:12.512657 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:12.569142  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:12.572458 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:12.583727  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:12.604269 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:12.616565  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:12.642760  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:12.658474 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:12.789796  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:33:12.922619 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:12.954697  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 21:33:12.968415  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:12.969439 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:12.971599  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:12.978705  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:12.991165 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:13.058552 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:13.182781  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:13.326179  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:13.327219  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:13.333369 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:13.359238 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:13.363602 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:13.375519  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:13.380125  9313 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.72:40467, user_credentials={real_user=slave}} blocked reactor thread for 44668.6us
I20240502 21:33:13.391791  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:13.484747  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:33:13.491907  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:33:13.524619 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:33:13.524784 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:13.687850  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 21:33:13.690603  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:13.691257  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:13.691354 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:13.712558 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:13.715425  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:13.742602 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:13.763402  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:13.767213  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:33:13.937031 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:13.961993  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 21:33:14.067245  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:14.078877  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:14.095072 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:14.095664  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:14.105612 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:14.118067  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:14.127305 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:14.156165  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:33:14.207291  9174 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.67:43203, user_credentials={real_user=slave}} blocked reactor thread for 50227us
I20240502 21:33:14.454159  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:14.471647  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:14.476876  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:14.500316 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:33:14.529793  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:33:14.530133 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:14.563869  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 21:33:14.573101 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:14.583187 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:14.647150  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:14.801291  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 21:33:14.862598  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:14.866931  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:14.881348 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:14.895913 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:14.897166  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:14.903419 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:14.919461  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:15.000025  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 21:33:15.086872 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:15.156150  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 21:33:15.186260 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:15.189738  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:15.204077  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:15.213137 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:15.227967  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:15.230857 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:15.236256  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:33:15.284315  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:33:15.401742  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 21:33:15.532403  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:15.538007 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:15.546526  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:15.556943  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:15.556921 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:15.562034 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:15.596983  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:15.694831 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:15.782078  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 21:33:15.935041  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:15.940135 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:15.961244  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:15.969302 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:15.979806  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:15.980834  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 21:33:15.981943  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 21:33:15.986558 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:15.994624  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:16.048327 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:16.060329 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 21:33:16.293284  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:16.299575  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:16.299692 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:16.315157  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:16.316754  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:16.320340 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:16.343667 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:16.560652 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 21:33:16.564460 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:16.647035  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:16.656972  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:33:16.668241  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 21:33:16.676133 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:16.679080 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:16.683272 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:16.684001  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:16.736474  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:16.816748  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 21:33:17.014459 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:17.016011  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:17.021775  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:17.023751  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:17.029289  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:17.035451 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:17.052305 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:17.096040 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:17.125147  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:33:17.204386  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:33:17.287571  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 21:33:17.355666  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:17.379292 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:17.397087  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:17.436223  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:17.471491  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:17.471637 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:17.477504 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:17.618207  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:33:17.641460 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:17.734423  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:17.755589  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 21:33:17.760432 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:17.770504  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:17.771713  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:17.794138 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:17.833886 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:17.896027  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:18.132762  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 21:33:18.154939  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:18.170150  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:18.171469 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:18.180001 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:33:18.192207 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:18.208527  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:18.212054 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:18.254129  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:18.354123  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 21:33:18.477821  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:18.484804 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:18.492326  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:18.514452 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:18.517271 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 21:33:18.528362 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:18.545756  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:18.546293  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:18.563587  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:33:18.609949 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:18.696341  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 21:33:18.852670  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:18.877179 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:18.886220 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:18.886358  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:18.887354 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:18.888679  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:18.944422  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 21:33:18.944841  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:19.136632 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:19.242161  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:19.261873  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:19.264608  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:19.269713 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:19.275962 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:19.277141  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 21:33:19.295632  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:19.304792 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:19.343065  9173 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 65921.9us
W20240502 21:33:19.344734  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:33:19.432332  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:33:19.474136  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:33:19.531515 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:19.559257  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:19.569571 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:19.592159  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:19.598348  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:19.619680  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:19.628928 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:19.653501 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:19.861027  9993 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:33:19.861250  9993 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:33:19.862020  9993 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:33:19.863775 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
W20240502 21:33:19.864439 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:19.864557 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:33:19.864813  9993 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 21:33:19.892848  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 21:33:19.905496  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:19.911262  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:19.940624 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:19.942381  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:19.976569 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:19.996879 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:20.026319 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:20.040614  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:20.070493  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:33:20.154697  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240502 21:33:20.310619  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:20.311475  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:20.318796  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:20.328819 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:20.329259 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:20.333704 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:20.364557  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:20.370863  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:33:20.446916 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:20.617619  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:20.620186 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:20.635331  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:20.646126  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:20.662317 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:20.663509 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:20.666719  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:20.676393  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 21:33:20.983966  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:20.997473 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:20.997913  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:20.999188 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 21:33:21.011320  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:21.018935 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:21.025533  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:21.025655 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:33:21.039386  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:33:21.044684  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:33:21.045706 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:21.315322  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:21.318425 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:21.319638 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:21.321367  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:21.331658  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:21.346444 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:21.367148  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 21:33:21.397053  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:21.569962  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:33:21.620218 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 21:33:21.621515 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:21.622210  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:21.646533 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:21.647992  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:21.649600 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:21.651147  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:21.659531  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:21.660066 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:21.692174  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:33:21.796763  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 21:33:21.995774  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:22.009003 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:22.012018  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:22.012921  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:22.013317 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:22.017791 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:22.032393 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:22.046617  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:22.199498  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 21:33:22.312356  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:22.323509 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:22.324941  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:22.329422  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:22.350771 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:22.359375 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:22.361958  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:22.426530 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:22.619860  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:33:22.654618  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 21:33:22.656703 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:22.657430  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:22.669515  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:22.702343 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:22.729840  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:22.737004  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 21:33:22.738970  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:22.795485 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:22.989679 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:23.076259  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:23.076342  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:23.082870 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:23.085995  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:23.086441  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:23.087074 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:23.098330 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:23.195387  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:33:23.387082 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 21:33:23.447535 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:23.448774  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:23.450899  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:23.459365  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:23.470770 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:23.492141  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 21:33:23.496330  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:23.518343 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:23.556542 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:23.789633  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:23.808852 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:33:23.819234  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 21:33:23.886895  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:23.904691  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:23.909632 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:23.940814  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:23.941998 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:24.003836  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:33:24.048499 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:24.148309  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:24.180117 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:33:24.186777  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 21:33:24.187494  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:24.191526 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:24.216964  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:24.222788  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:24.235209 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:24.288859  9173 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 48818.7us
W20240502 21:33:24.303707  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:33:24.310976  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:33:24.471668 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:24.521270  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:24.521600 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:24.525624  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:24.528095  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:24.530926 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:24.549443 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:24.589218  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:24.743016  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:33:24.747707  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 21:33:24.850631  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:24.869724 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:24.870472  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:24.872215 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:24.876430  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:24.877722 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:24.892448  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:24.894078 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:25.181641  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:25.183521 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:25.188107  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:25.210968 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:25.219444  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:25.232846 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:25.235427  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:25.238514  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 21:33:25.243469  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:33:25.267624  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:33:25.317138  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:33:25.419333 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:25.514041 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:25.516700  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:25.517716  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:25.527019 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:25.539527  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:25.552892 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:25.566147  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:25.792421 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 21:33:25.829663  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:25.859078 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:25.859320  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:25.877883  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:25.882158 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:25.892928 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:25.903172 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:25.928512  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:26.021418  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:33:26.106660  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:33:26.109934  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 21:33:26.189389  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:26.204836 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:26.218971  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:26.230026  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:26.250897 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:26.256455 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:26.278151  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:26.293568 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:26.366237  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 21:33:26.566442 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:26.566488  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:26.577266  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:26.583998 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:26.617610 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:26.629317  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:26.654852  9038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.71:35857, user_credentials={real_user=slave}} blocked reactor thread for 72544.5us
I20240502 21:33:26.680977  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:26.741149  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:33:26.828150 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:33:26.851064 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:26.871450  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 21:33:26.988113 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:26.990280  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:27.016288  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:27.023931 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:27.035450  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:27.036664  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:27.063071 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:27.072309  9173 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 64995.9us
W20240502 21:33:27.076721  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:33:27.298581 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:27.349576  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:27.352480 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:27.360016  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:27.361167 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:27.380894  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:27.410072 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:27.424362  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:27.518819  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:33:27.680088  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:33:27.686401  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 21:33:27.688660  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:27.701826 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:27.725931  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:27.730963  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:27.754844 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:27.798689 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:27.806425 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:27.816735  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:27.909678  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 21:33:28.065712  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:28.070952  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:28.073705 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:28.073865 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:28.095741 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:28.096050  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:28.116400  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:28.266111 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:33:28.292353 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:28.441565  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:28.444444  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:28.450055 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:28.472278 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:28.473310  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:28.480495 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:28.481999  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:28.626657  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:33:28.671738  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:33:28.742012  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:33:28.781039 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:28.792084  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:28.806999  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:28.816468 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:28.836306 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:28.860581  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:28.868010 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:28.884975  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:28.918299  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:33:29.085969  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 21:33:29.089278  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 21:33:29.155359  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:29.160329  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:29.162088 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:29.165709 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:29.166098  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:29.230520 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:29.251488  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:29.344377 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:29.476002  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:29.490937  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:29.501330 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:29.511713 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:29.514616  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:29.518965  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:29.525348 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:29.827986  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240502 21:33:29.834966  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:33:29.866142 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:29.872432  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:29.880251  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:29.881003 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:29.881038  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:29.892984 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:29.904201 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:29.946107  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:30.027272  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:33:30.061625  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 21:33:30.206351 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:30.210053  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:30.212409 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:30.229053  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:30.243070 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:30.245276  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 21:33:30.261747  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:30.308213  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:30.375317 11295 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.68:40931, user_credentials={real_user=slave}} blocked reactor thread for 47212.2us
W20240502 21:33:30.388376 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:30.400002  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:33:30.496299  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 21:33:30.514770  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:30.526541  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:30.528589 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:30.536815  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:30.544389 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:30.545768 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:30.546419  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:30.547874  9993 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:33:30.548130  9993 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:33:30.549204  9993 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:33:30.557821 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:30.561282 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
I20240502 21:33:30.561494 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:33:30.561810  9993 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 21:33:30.616479  9038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.68:40931, user_credentials={real_user=slave}} blocked reactor thread for 48192.5us
I20240502 21:33:30.893069  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:30.901455  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:30.903151 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:30.909569  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:30.914500 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:30.936829  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:30.945341 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:33:30.987448 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:31.041231  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:33:31.050513  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:33:31.144307 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 21:33:31.208822  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:31.239655  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:31.240324 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:31.276930  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:31.318732 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:31.325094  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 21:33:31.373898 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:31.395102  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:31.483268 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:31.505533  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:33:31.521870  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:33:31.618006  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 21:33:31.646374 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:31.655092  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:31.667552  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:31.696502  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:31.703799 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:31.716393  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:31.723948 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:31.984936  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:31.996989  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:33:32.000367 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:33:32.006120 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:32.021795 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:32.025388 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:32.025923 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:32.034546  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:32.047437  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:32.344480  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:32.348527 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:32.356770  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:32.357616  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:32.362958 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:32.372061 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:32.408160  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 21:33:32.416069  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:32.484129 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:32.605221  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 21:33:32.669888 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:32.681341  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:32.686678 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:32.690182  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:33:32.697038  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 21:33:32.697917 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:32.703788  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:32.703951  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:32.895534  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:33:32.913425 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:33.028136 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:33.034550  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:33.041358 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:33.043313  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:33.051383  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:33.058207  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:33.094022  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 21:33:33.108111 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:33.228899  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:33:33.336990 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:33.432559  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:33.447049 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:33.454432 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:33.467687  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:33.473582  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:33.604774  9993 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:33:33.605031  9993 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:33:33.605914  9993 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:33:33.607975 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
W20240502 21:33:33.608613 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:33.608762 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:33:33.609555  9993 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 21:33:33.629931  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:33.646106 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:33.684510  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:33:33.710585  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:33:33.721720  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 21:33:33.851675  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:33.857167 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:33.862023  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:33.863724 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:33.864233 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:33.864588 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:33.866039  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:33.869472 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 21:33:33.878707  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:34.092032  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 21:33:34.120913  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:34.146970 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:34.150897  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:34.154117  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:34.178010  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 21:33:34.185503 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:34.205132 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:34.219590  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:34.384183 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:34.468737  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:34.476056  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:34.476056  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:34.476194 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:34.508697 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:34.511554  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:34.531209 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:34.619288  9173 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 87865.8us
I20240502 21:33:34.859944  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:34.862210 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:34.862754  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:34.869236  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:34.885762 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:34.893150 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:34.894338 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:34.907562  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:34.989482  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:33:35.111778  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:33:35.113778  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 21:33:35.202313  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:35.219651 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:35.229007  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:35.229493  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:35.230548 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:35.289433 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:35.291901  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 21:33:35.295536  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:35.367614  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:33:35.394160 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:35.515292  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240502 21:33:35.519631  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 21:33:35.542637  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:35.578361 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:35.581709  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:35.591248  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:35.624234 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:35.635965  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 21:33:35.638672  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:35.647863 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:35.947896  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:33:35.962468 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:35.975250 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 21:33:36.020701 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:36.023635  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:36.025518  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:36.060477 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:36.060669  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:36.068053 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:36.126289  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:36.164824  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:33:36.270169  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 21:33:36.374966 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:36.375996  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:36.383651  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:36.399029 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:36.413411  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:36.414340 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:36.415414  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:36.460281 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:36.514673  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:33:36.671599  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 21:33:36.715520 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:36.725050  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:36.727097  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:36.727710  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:36.738948 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:36.766762  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:36.767809 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:36.921171 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:37.064759 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 21:33:37.077852  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:37.078030 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:37.087837  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:37.098074 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:37.101740  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:37.106622 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:37.126245  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:37.329480  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 21:33:37.437664  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:37.440457  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:37.441752  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:37.443006 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:37.464349 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:37.465384  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:37.478099 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:37.510943 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:37.813218 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:37.833385  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:37.837553  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:37.846483 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:37.865159  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:37.869802  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:37.880303  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:33:37.905807  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 21:33:37.906690 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:37.921586  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:33:37.949090 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:38.174841 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:38.183396  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:38.186203  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:38.190335  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:38.194923 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:38.199668  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:33:38.204527  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 21:33:38.213657 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:38.242799  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:33:38.254451  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:33:38.343966 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:38.492272  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 21:33:38.537572  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:33:38.547739 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 21:33:38.547993  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:38.549269  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:38.552310 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:38.573215 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:38.606279 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:38.634058  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:38.674593  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:33:38.871819 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:38.894790 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:38.898566  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:38.899995  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:38.927475 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:38.930291  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:38.962376 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:38.967712  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:39.022272  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:33:39.154899  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 21:33:39.284605  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:39.295621  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:39.320860  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:39.371997 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:39.390774 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:39.397801  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:39.480808 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:39.487425 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:39.727139  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 21:33:39.732088  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:39.736727 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:39.737226  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:39.752377 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:39.755230  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:39.782610 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:39.810259  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:40.062136 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:40.128726  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:40.144130 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:40.156772 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:40.162714 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:40.180548  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:40.212772  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:40.241704  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 8 similar messages]
I20240502 21:33:40.256649  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:40.301411  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:33:40.512236  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 21:33:40.523906  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:40.526706 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:40.534479  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:40.537073 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:40.541590 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:40.543889  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240502 21:33:40.548020 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:40.564863  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:40.564859  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:40.692224  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:33:40.698539  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:33:40.763021  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:33:40.831377  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 21:33:40.868503  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:40.869735 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:40.888731 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:40.894564  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:40.896425  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:40.919545  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:40.924404 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:41.050644  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:33:41.068328  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:33:41.092969 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:41.165450  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:41.186542  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:41.190750 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:41.191658 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:41.192098 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:41.194804  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:33:41.203805  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 21:33:41.205462  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:41.208770 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 21:33:41.514371  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:41.527652  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:41.535728 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:41.548645  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:41.551545 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:41.568217  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:41.568675 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:41.630338 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:41.650743  9038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.68:40931, user_credentials={real_user=slave}} blocked reactor thread for 52452.7us
I20240502 21:33:41.891759  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:41.895987 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:41.902549  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:41.905470 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:33:41.908859  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 21:33:41.909901 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:41.910571  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:41.915199  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:42.128414 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 21:33:42.130156 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:42.229460  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:42.232301 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:42.244168  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:42.247040 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:42.253011  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:42.272251 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:42.275501  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:42.386013  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:33:42.474336  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 21:33:42.600416  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:33:42.605935 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:42.621340 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:42.623348  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:42.668212 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:42.672649 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:42.674984  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:42.675455  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:42.946501  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 21:33:43.016347  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:43.019321  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 21:33:43.019482 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:43.026269  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:43.033938  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:43.057379 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:43.072000 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:43.084020  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:43.195801  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:33:43.209147 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:43.242944  9993 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:33:43.243196  9993 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:33:43.244184  9993 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:33:43.246906 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
W20240502 21:33:43.249244 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:43.249419 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:33:43.249719  9993 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 21:33:43.292660  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:33:43.341699  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:33:43.363375  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 21:33:43.428514  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:43.436771  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:43.453435 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:43.458125  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:43.459424 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:43.464046 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:43.497877  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:43.507004  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:33:43.727324 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:33:43.750012 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:43.767689  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:43.770366 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:43.775259 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:43.779903  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:43.798480  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:43.804617  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:43.809013 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:43.833108  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:33:43.835270  9445 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.69:42049, user_credentials={real_user=slave}} blocked reactor thread for 51057.8us
I20240502 21:33:44.092703  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:44.100768  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:44.106804 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:44.140098 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:44.143043 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:44.143448  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:44.179601 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:44.201498  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:44.225497  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 21:33:44.484859  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:44.486616  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:44.492954  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:44.493330  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:44.493399 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:44.495363 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:44.519490 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:33:44.685581 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:44.800097  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:44.815538  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:44.818023 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:44.819399  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:44.822108 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:44.852650  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:44.856601 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:45.026038  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 21:33:45.141881 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:45.142122 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:45.142565  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:45.143560  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:45.154763  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:45.163967  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 21:33:45.178859 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:33:45.204934 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:45.232062  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:45.376245  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 21:33:45.487830  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:45.492916  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:45.495013 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:45.505242  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:45.506984  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:45.518159 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:45.530319 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:45.560091  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 21:33:45.622215  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:33:45.676012  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:33:45.772037 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:45.777845  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 21:33:45.873965  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:33:45.878799  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 21:33:45.888775  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:45.889228  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:45.889386 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:45.916574 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:45.962085 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:45.990170  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:33:46.090843 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 21:33:46.099285  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:46.125370  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:33:46.153577  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:33:46.153733  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:33:46.168905 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:46.283396  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:46.284373  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:46.290205  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:46.291855 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:46.301676 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:46.314317  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:46.333319 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:46.574574  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:46.576570  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:46.602594 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:46.606607  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:46.609779 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:46.622000  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:46.623383 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:46.658087 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:46.903077  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:46.909531 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:46.913921  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:46.915513  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:46.939854 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:46.941409  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:46.943308 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:46.991320  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:33:47.123871 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:47.225508  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:47.228480 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 21:33:47.245047  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:47.245206 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:47.255506  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:47.273452 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:47.346174 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:47.381253  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:47.464282  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:33:47.485232  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 21:33:47.576721 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:47.579998 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:47.587654  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:47.589162  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:47.608199  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:47.615911 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:47.667402 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:47.676982  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:47.957300  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:47.967326  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:47.976339 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:47.978593 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:47.987903  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:47.987923  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:48.022929 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:48.071995  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:33:48.079852 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:48.195211  9993 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:33:48.195624  9993 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:33:48.197348  9993 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:33:48.200340 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
W20240502 21:33:48.204121 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:48.204273 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:33:48.204989  9993 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 21:33:48.217486  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:33:48.224925  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:33:48.318918  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 21:33:48.344659  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:48.351573  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:48.353092  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:48.355139 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:33:48.376614  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 21:33:48.385965 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:48.423084 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:48.468818  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:48.564426  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:33:48.622714 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:48.652247  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 21:33:48.743964  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:48.749855  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:48.762626 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:48.767364 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 21:33:48.775293 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:48.776607  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:48.779505 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:48.787238  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 21:33:48.811475  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:49.045815 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:49.084074 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:49.100257  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:49.106552  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:49.123502 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:49.126577 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:49.129576  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:49.174638  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:49.237761  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 37 similar messages]
I20240502 21:33:49.494946  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:49.501754  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:49.522686 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:49.523519  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:49.526726  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:49.527832 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:49.536995 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:49.549980 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:49.822132  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 21:33:49.846379  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:49.865141 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:49.876471  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:49.891010  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:49.894356 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:49.904073  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:49.916543 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:50.064466 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:50.177728  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:50.184319  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 21:33:50.186776 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:50.206346  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:50.211189 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:50.214866 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:50.215358  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:50.215392  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:50.486369  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:33:50.506755  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:33:50.508355  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:33:50.523801 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:50.526886  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:50.548282 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:50.556965 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:50.557669  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:50.557875  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:50.566511 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:50.591526  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:50.704682  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:33:50.806025  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240502 21:33:50.862429  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 21:33:50.869316 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:33:50.871793  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 21:33:50.875356  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:50.890323  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:50.907976 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:50.914649 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:50.928622  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:33:50.948181 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:50.948644  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:50.958220  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:33:51.001001 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:33:51.023197  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 21:33:51.220927  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:51.223680 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:51.270984 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:51.284049  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:51.289309  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:51.293745  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:51.310756 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:51.321951 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:51.329456  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 21:33:51.551662 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:51.552687  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:51.554922  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:51.565457 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:51.568231  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:51.581933 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:51.596105  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:51.721926 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:51.903997  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:51.921342 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:51.923593  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:51.932117 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:51.932950 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:51.978600  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:52.001379  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:52.174165  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:33:52.231240 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 21:33:52.234599 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:52.235356  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:52.256287 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:52.283819  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:52.284693  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:52.298554  9993 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:33:52.298761  9993 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:33:52.299520  9993 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:33:52.306810 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
W20240502 21:33:52.309619 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:52.309814 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:33:52.310101  9993 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 21:33:52.314793 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:52.326961  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:52.350244 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:52.440780  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:33:52.526013  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 21:33:52.604542  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:52.608098 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:52.623234  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:52.650717  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:52.669237 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:52.681491 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:52.683614 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:52.701759  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:52.820290  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 21:33:52.930270  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:52.931205  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:52.941993  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:52.966459 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:52.970456  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:52.993434 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:53.021493 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:53.125300 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:53.157423  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:33:53.262652  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 21:33:53.291381 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:53.301102 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:53.301647  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:53.304514  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:53.304885  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:53.306159  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:53.309671 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:33:53.404978  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:33:53.406333 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:33:53.448581  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:33:53.469537  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:33:53.609117  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 21:33:53.610276  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:53.623272  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:53.627468 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:53.668291  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:53.672551 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:53.684142 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:53.719228  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:53.741967 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:53.841912  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 21:33:54.006552 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:54.007496  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:54.028568 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:54.030894  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:54.033991  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:54.036979 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:54.051052  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:54.226019 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:54.275319  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 21:33:54.388064  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:54.388828 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:54.408850  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:54.429145  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:54.439155 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:54.445533  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:54.493244 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:54.740595  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:54.741964 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:54.747946 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:54.750972  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:54.777200 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:54.779322  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:54.830405  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:54.831540 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:55.086642  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:33:55.095166  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 21:33:55.193512 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:55.205541  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:55.216631  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:55.219902 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:55.220116  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:55.231058 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:55.251919  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:55.273294  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:33:55.316748 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:55.443863  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:33:55.467900  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 21:33:55.570161 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:55.570614  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:55.572553  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:55.620607 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:55.635169  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:55.637624 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:55.638468  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:55.695400 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:55.715849  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:33:55.772195  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:33:55.889864  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 21:33:55.894325 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:33:55.900593  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 21:33:55.927829  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:55.933944 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:55.943238  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:55.950811 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:55.976442  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 21:33:55.977475  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:56.007088  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 21:33:56.009124 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:56.013664  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:56.276790 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:56.277680  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:56.280133  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 21:33:56.286080 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:56.288059  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:56.300493 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:56.306875  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:56.313392  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:56.341341 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:56.608922  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:56.620373 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:56.635358  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:56.647616 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:56.651748  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:56.664359 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:56.675973  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:56.715500 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:56.998874  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:57.011225 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:57.011978 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:57.014751  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:57.015986 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:57.029274  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:57.046077  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:57.101140 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:57.346433  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:57.353081 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:57.372563  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:57.378408  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 21:33:57.396943 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:57.423753  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 21:33:57.424091 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:57.444218  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:57.444885  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:57.537470 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 21:33:57.568532 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:57.685565  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:33:57.794224  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 21:33:57.797865  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:57.800756 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:57.810815  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:57.811404 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:57.815752 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:57.819685  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:57.826360  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 21:33:57.868796  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:33:57.940189  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:33:58.054787 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:58.130507  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 21:33:58.177055  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:58.183562 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:58.205838  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:58.214509 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:58.247951 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:58.265866  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:58.353144  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:58.425935  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:33:58.508667 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:33:58.525873 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 21:33:58.553275  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:58.561640 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:33:58.578444  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 21:33:58.582453  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:58.615232  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:58.643173  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:58.644636 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:58.647202 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:58.649708  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:33:58.836794  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 21:33:58.880986 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:58.883062  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:58.914312  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:58.923417 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:58.931602  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:58.958067 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:58.972107  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:59.045841 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:59.265838 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:59.267565  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:59.290793  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:59.301329  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:59.303843 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:59.328219  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 21:33:59.342044 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:59.344949  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:33:59.544992 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:33:59.628718  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:59.642733 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:33:59.646054  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:33:59.710433  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:33:59.722887 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:33:59.741447  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:33:59.745167 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:33:59.853991  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:33:59.984261 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:00.020226  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:00.029428 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:00.041190  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:00.042800 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:00.045506  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:00.062323  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:00.064913 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:00.136948  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:34:00.305543  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:34:00.339166  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 21:34:00.385136  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:00.419484  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:00.459792 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:00.461941  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:00.527585 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:00.643371 11025 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.71:35857, user_credentials={real_user=slave}} blocked reactor thread for 60466.3us
I20240502 21:34:00.646849 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:34:00.723125  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 21:34:00.729732  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:00.733599  9444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.71:35857, user_credentials={real_user=slave}} blocked reactor thread for 241576us
W20240502 21:34:00.769424 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:00.788236 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:34:00.791054  9173 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 305678us
W20240502 21:34:00.795400  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:34:00.875428  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:34:01.024096  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:34:01.121946 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:01.133944  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 21:34:01.238133  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:34:01.296640  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:34:01.393410  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:34:01.553229 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:01.673940 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:01.701812  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:01.719519  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:01.748021  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:01.767426 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:01.771618 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:01.796871  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:02.063041  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:02.084401 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:02.085937  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:02.092800 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:02.109845  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:02.110188  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:02.148515 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:02.178545 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:02.365869  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 21:34:02.434620  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:02.435405 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:02.441175  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:02.471347  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:02.473973  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:02.475299 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:02.476827 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:02.624461  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:34:02.637544  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:34:02.679715 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:34:02.744846 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:02.746668 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:02.762018  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:02.774055 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:02.782685  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:02.813884  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:02.856850  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 21:34:02.874900 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:02.900660  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:02.907968  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:34:03.020824  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:34:03.065867  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 21:34:03.102931  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:03.118371  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:03.125725 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:34:03.127566 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 21:34:03.130178  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:03.132418 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:03.146952  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:03.156266 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:03.182566 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:03.371613  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:34:03.404126  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 21:34:03.484580  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:03.490684 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:03.492869  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:03.505221  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:03.519716 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:03.523985  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:03.524477 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:03.591094 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:03.795763  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 21:34:03.833752  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:03.856123 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:03.862975  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:03.882822 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:03.915585  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:03.916980  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:03.918324 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:34:03.977072  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:34:04.109285 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:04.184435  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:04.211992 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:04.230304 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:04.230656  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:04.231932  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:04.270988  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:04.292418 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:04.380159  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 21:34:04.587394  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:04.602849  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:04.617270 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:04.624163 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:34:04.631099 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:04.633008  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:04.636404 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:04.640676  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:04.720749  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 21:34:04.960458  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:05.004168 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:05.011756  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:05.014494  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 21:34:05.019023  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:05.019706 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:05.027730  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:05.056941 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:05.153506  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:34:05.181840 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:05.280884 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 21:34:05.336328  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:05.336337  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:05.345687 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:05.371829  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 21:34:05.373693 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:05.389963 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:05.427462  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:05.448583  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:05.453960  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:34:05.616683  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:34:05.680164 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:05.729049  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:05.737809 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:05.743892  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:05.745193  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:05.745810 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:05.764534  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:05.794270 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:06.030838  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 21:34:06.042531  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:06.060753 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:06.067646  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:06.070546 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:06.079538 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:06.081283  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 21:34:06.086086  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:06.091320  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:06.152576  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 21:34:06.205180 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:06.386081  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:06.394342  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 21:34:06.399030 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:06.423662  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:06.427496  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:06.461166 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:06.467135 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:06.471000  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:06.478024  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:34:06.610684  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:34:06.687688 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:06.713608  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:06.715325 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:06.722729  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:06.737825  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:06.739889 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:06.757444  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:06.793234 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:07.015321  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:07.048808  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:07.053053 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:07.081686 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:07.089324  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:07.090749 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:07.095373  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:34:07.095775  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:34:07.146845 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:07.354547  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 21:34:07.375494  9993 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:34:07.375700  9993 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:34:07.376654  9993 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:34:07.379354 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
W20240502 21:34:07.379583 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:07.379720 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:34:07.380174  9993 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 21:34:07.393172  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 21:34:07.444898  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:07.445842 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:07.447641 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:07.456419  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:07.474804  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:07.500394 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:07.550454  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:34:07.614068 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:07.625267 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:34:07.637655  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:34:07.638868  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 21:34:07.802841 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:07.805501  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:07.806430  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:07.808451  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:07.808745  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:07.820003 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:07.834688 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:07.888835  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:34:08.029392 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 23 similar messages]
W20240502 21:34:08.073812 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:08.167385  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:08.170495 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:08.196080  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:08.212509 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:08.241253  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:08.242385  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 21:34:08.247731 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:08.280448  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:08.326118  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:34:08.540521  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 21:34:08.573345  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:08.579367  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:08.613127 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:08.614470 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:08.645226 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:34:08.646795 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:08.675127  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:08.675283  9039 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.69:42049, user_credentials={real_user=slave}} blocked reactor thread for 83597.7us
I20240502 21:34:08.741698  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:08.760246  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 21:34:09.021845  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:09.023180  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:09.034870  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:09.041561 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:09.053524 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:09.091859 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:09.130092 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:09.158165  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:09.242558  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 21:34:09.403476  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:09.407691 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:09.411250  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:09.444190  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:09.453450 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:09.486733  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:09.516614  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 21:34:09.561970 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:09.565305 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:09.594276  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 21:34:09.713167 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:09.713310  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:09.729265  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:09.747812 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:09.749652 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:09.770561  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:09.775295  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:09.882382  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:34:09.893630  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:34:09.916626 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:34:10.059680 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:10.100651  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:10.111364 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:10.113970 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:10.132015  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:34:10.141538  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 21:34:10.152820  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:10.160914  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:10.218840 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:10.397017  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 21:34:10.451788  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:10.460795  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:10.477955  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:10.486495 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:10.489868 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:10.513804 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:34:10.526540 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:10.546981  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:10.706521  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:34:10.763352  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 21:34:10.856014  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:10.864367 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:10.882688 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:10.891749  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:10.912868 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:10.924521  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:10.930932  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 21:34:10.980751  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:11.081168  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:34:11.131055 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:11.173193  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240502 21:34:11.236851  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:11.241990  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:11.244643 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:11.245512  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:11.250247 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:11.310066  9444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.65:42177, user_credentials={real_user=slave}} blocked reactor thread for 100429us
I20240502 21:34:11.348475 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:11.353557  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:11.591924  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:11.611821 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:11.631039  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:11.635097 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:11.636883  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:11.637043 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:11.652046  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 21:34:11.663364  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:11.687299 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:11.802120  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:34:11.857285  8946 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:11.858428  8946 catalog_manager.cc:4573] TS 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4d1a31513bb64578a85adf1f622e5bc0: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:11.949764  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:11.955343  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:11.956147 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:11.958750  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:11.991928 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:12.016301  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:12.047264 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:12.151952 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:12.244172  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 21:34:12.271363  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:12.276783  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:12.297271 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:12.301481 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:12.318212  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:12.333307 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:12.333765  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:12.398620  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:34:12.420202  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:34:12.424770 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:34:12.602917  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 21:34:12.634702  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:12.637409 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:12.653935  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:12.676647 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:12.678083 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:12.684460 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:12.698407  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:12.714690  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:12.862020  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:34:12.863150  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:34:12.936774  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 21:34:12.994870  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:13.016012  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:13.034286  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:13.071676 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:13.081528  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:13.084825 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:13.120569 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:34:13.181946 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:34:13.182907 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:13.472584  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:13.480113 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:13.488515  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:13.494799  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:13.519688  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:13.556099 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:13.570888  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:34:13.583355  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 21:34:13.588111 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:13.646998  9173 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 50332.3us
W20240502 21:34:13.660652  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:34:13.699030 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:13.717986  9173 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 57355.8us
I20240502 21:34:13.873634 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:13.878290  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:13.886873  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:13.898214  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:13.906688 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:13.907193  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:13.928334 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:13.947578  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 21:34:14.212278  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:14.217339 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:14.229295  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:14.230242  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:14.271697 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:14.276659 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:14.278882  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:14.297719 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:14.596225  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:14.598909  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:14.608860 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:14.630432  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:14.637064 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:14.637548  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 21:34:14.641546  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:34:14.676437  9172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.68:40931, user_credentials={real_user=slave}} blocked reactor thread for 97687.7us
W20240502 21:34:14.689504 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:14.721334  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 21:34:14.734223  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:14.743391 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:14.868603  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 21:34:14.991909  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:14.996464  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:14.997295 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:15.012984  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:15.012923 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:15.062781 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:15.093750 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 21:34:15.105469  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:15.235502 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:15.394546  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 21:34:15.423173  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:15.453366  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:15.453444 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:15.456166 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:15.460603  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:15.472123 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:15.477473  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:15.483773  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:34:15.646726  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:34:15.778703 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:15.789829  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:15.806450  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:15.815701 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:15.817350  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:15.818122 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:15.911736 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:15.917117  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:16.091423  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:34:16.099387  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:34:16.163434 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:16.215986 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:16.218554  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:16.229950  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 21:34:16.235244  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:16.251315 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:16.251859  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:34:16.279902  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 21:34:16.312779 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:16.326040  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:16.532027  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 21:34:16.569803  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:16.577247  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:16.589613  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:16.593354 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:16.618681 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:16.623482 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:16.628429  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:16.672286 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:16.914875 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:16.921590  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:16.921816  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 21:34:16.924966  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:16.936233  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:16.945058  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:16.964485 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:16.967684 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:17.096035  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:34:17.117594 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:17.273859  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:17.282814  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:17.283365 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:17.290649 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:17.293503 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:17.294431  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:17.298188  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 21:34:17.315856  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:17.566769  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:17.567080 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:17.568113  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:17.584448 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:17.590973 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:17.597865  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:17.617497  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:17.618616 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:17.634436  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:34:17.760985  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:34:17.794435  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 21:34:17.893867  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:17.899822  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:17.907728 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:17.911178 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:17.911724  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:17.919751 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:17.955808  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:17.963557 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:34:18.175197  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:34:18.179198  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:34:18.236729 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 21:34:18.239794 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:18.327757  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:18.328269  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:18.335565  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:18.337253  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:18.348413 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:18.357308 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:18.405324 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:18.475147  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:34:18.689498  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:34:18.704680  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 21:34:18.717936 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:18.732319  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:18.733552  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:18.736239 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:18.739596  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:18.775653 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:18.777150 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:18.828382  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:18.899366  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 21:34:19.083562  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:19.096107  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:19.098212  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:19.100453 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:19.101502 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:19.102246  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:19.113248 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:34:19.194939  9174 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.67:43203, user_credentials={real_user=slave}} blocked reactor thread for 48583.5us
W20240502 21:34:19.239027  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:34:19.390228 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:19.458422  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:19.461340 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:19.474995  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:19.496184 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:19.504284 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:19.520727  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:19.521718  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:19.565033  9310 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.71:35857, user_credentials={real_user=slave}} blocked reactor thread for 95337.5us
W20240502 21:34:19.678669  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240502 21:34:19.698499  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 21:34:19.853238  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:19.856774 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:19.859822  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:34:19.880784 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:19.890111 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:19.896281  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:19.914706 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:19.952066  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:19.954703  9038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.71:35857, user_credentials={real_user=slave}} blocked reactor thread for 78985.8us
W20240502 21:34:20.069615  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:34:20.199980  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 21:34:20.230636  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:20.245726 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:20.256242 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:20.274601  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:20.276481  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:20.292750 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:20.293336  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:20.373823 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:34:20.376499  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:34:20.461745 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:20.603607  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:20.623137  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:20.625960 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:34:20.628916  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 21:34:20.654321 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:20.669958  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:20.675611 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:20.681547  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:20.895016 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:21.028147  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:21.028811 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:21.036340  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:21.039853  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:21.050609  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:21.052101 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:21.053995 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:21.157752  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:34:21.245755  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:34:21.347751  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 21:34:21.369163  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:21.371351  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:21.392642 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:21.395305  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:21.398689 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:21.416787 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:21.427309  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:21.469861  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:34:21.475581 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:21.498135  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 21:34:21.668648  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 21:34:21.714061  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:21.717985  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:21.738902  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:21.752899 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:21.811915 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:21.830724  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:21.875380 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:34:21.982439 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:22.041744  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:22.054206  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:22.057529 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:22.065579 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:22.077275  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:22.077603 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:22.087296  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:22.132823  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:34:22.193173  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:34:22.412928 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:22.415304  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:22.419291 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:22.426335  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:22.458027  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:22.458676 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:22.465602  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 21:34:22.467909  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:22.477458 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:22.717051 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 21:34:22.740235 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:22.746912  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:22.749301  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:22.770136 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:22.773727  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:22.841591 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:22.845412  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:22.860347 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:22.882004  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:34:23.093497  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 21:34:23.105234 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:23.108624  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:23.145077  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:23.151361 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:23.155381  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:23.203693  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:34:23.205152  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 21:34:23.219491 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:23.223801  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:23.281668 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:34:23.283140 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:23.479918  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 21:34:23.554715  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:23.562258  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:23.572165 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:23.572352  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:23.591571 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:23.593336 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:23.613189  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:23.642153  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:34:23.735827 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:23.770138  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:34:23.825874  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 21:34:23.909060  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:23.915647  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:23.931097 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:23.939466 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:23.939668 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:23.958703  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:23.967088  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:24.164391 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:24.168884  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 21:34:24.272996  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:24.273192 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:24.278723  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:24.334831  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:24.365809 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:24.387502 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:24.391755  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:24.574569  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:34:24.591791 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:24.664995  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:24.683588 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:24.689999 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:24.706365  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:24.717442  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:24.732084  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:24.755643 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:24.838449  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 21:34:24.852756  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:34:25.011869 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:25.061618  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:25.065593 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:25.092025  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:25.098589 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:25.108999  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:25.122570 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:25.124424  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:25.282588  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:34:25.282855 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 21:34:25.479036 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:25.480192  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:25.480273  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:25.498112 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:25.512001 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:25.530504  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:25.531163  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:25.593503 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:25.599625  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 21:34:25.608065 19349 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:34:25.608275 19349 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:34:25.609225 19349 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:34:25.610957 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
W20240502 21:34:25.613147 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:25.613286 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:34:25.613845 19349 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 21:34:25.816850  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:25.819716 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:25.835716  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:25.860774 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:25.863031  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:25.889605  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:25.925415 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:25.950682  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:34:26.068432  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 21:34:26.176419  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:34:26.182602 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:26.192816  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:26.193419 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:26.218111 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:26.227010  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 21:34:26.231438 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:26.287859  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:26.291960  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:34:26.292251  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 21:34:26.350970  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:26.461022  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 21:34:26.560434  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:26.564002 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:26.566475  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:26.567848 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:26.567931  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:26.579947  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:26.585345 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:26.648950 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:26.801084  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240502 21:34:26.802105  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 21:34:26.880846 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:26.883623  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:26.888052  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:26.931548 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:26.933019  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:26.946583 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:26.970609  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:27.173322 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:27.206543  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:27.238750 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:27.245760  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:27.251745  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:27.258500 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:27.276507 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:27.333092  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 21:34:27.353096  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:27.454963  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 21:34:27.594404  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:27.601001  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:27.605165 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:27.613803 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:27.615523  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:27.636289 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:27.637773  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:27.750924 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:27.769600  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:34:27.860164 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 21:34:27.952437  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:27.955449  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:27.969583  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:27.970152 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:27.994833 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:28.010972  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:28.016780 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:28.023151  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:34:28.220764 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:28.319882  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:28.335615  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:34:28.345883 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 21:34:28.371361  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:28.375134 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:28.378116  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:28.383750 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:28.396669 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:28.421875  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:34:28.424714  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:34:28.426725  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:34:28.636461 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:28.659962  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 21:34:28.709870  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:28.715389  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:28.716663 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:28.721271 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:28.736066  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:28.750927  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:28.778659 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:28.880882  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 21:34:29.056128  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:29.058096 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:29.058581  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:29.085397  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:29.089359 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:29.093019 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:29.101780  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:34:29.104327  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 21:34:29.121912  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:29.259620 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:29.278167  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 21:34:29.442822  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:29.457141 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:29.457861  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:29.477820 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:29.478298 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:29.478297  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:29.480001  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:29.702623  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:34:29.754798 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:29.762014  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:29.770527  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:29.781193 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:29.798583  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:29.805562 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:29.819331 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:29.837369  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:29.932862  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 21:34:30.108748 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:30.109180  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:30.118849  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:30.120837 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:30.124469  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:30.141130 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:30.161222  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:30.293486  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:34:30.315958 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:30.387887  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:34:30.422911 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 21:34:30.485241  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:30.499545  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:30.500648 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:30.505965 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:30.508312 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:30.515461  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:30.559489  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:30.648293  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:34:30.863656 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:30.881814  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:30.898037 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:30.898978 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:30.900173  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 21:34:30.902194  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:30.924032  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:30.935730 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:30.997236  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:31.221938  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:34:31.235592  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 21:34:31.254040  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:31.258285 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:31.284914  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:31.286815 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:31.285612 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:31.305909  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:31.311833 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:31.453593  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 21:34:31.575150  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:31.588980  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:31.595242  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 21:34:31.596392 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:31.602365 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:31.604840  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:31.613794 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:31.617954  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:31.688822  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:34:31.702621 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:31.704164  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 21:34:31.918861  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:31.937405 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:31.956482  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:31.958106 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:31.959450  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:31.962182 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:31.973695  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:34:32.009362  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 21:34:32.120946  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:34:32.138029 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:32.238590  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:32.269487 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:32.295918  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:32.319456  9444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.71:35857, user_credentials={real_user=slave}} blocked reactor thread for 69664.5us
I20240502 21:34:32.323314 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:32.334477  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:32.370723 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:32.396711  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:32.491900  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:34:32.593411 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:32.600209  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:32.605273 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:32.617441 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:32.619277  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:32.640271 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:32.641140  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:32.663718  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:32.665838  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:34:32.888227  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 21:34:32.906991 20374 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:34:32.907241 20374 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:34:32.908111 20374 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:34:32.910189 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
W20240502 21:34:32.911159 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:32.911335 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:34:32.911640 20374 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 21:34:32.977455  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:32.985181 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:32.992857  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:33.008777 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:33.021613 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:33.027777  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:33.055783 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:33.059722 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 21:34:33.073732  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:33.143721  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 21:34:33.302634  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:33.305270  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:33.320904  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:33.321331 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:33.330775 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:33.344656 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:33.358321  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:33.452988  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:34:33.509969 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:34:33.605818 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:33.607496  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:33.612437 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:33.624019  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:33.627148 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:33.642058 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:33.658804  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:33.663372  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:33.685860  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:34:33.698144  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 21:34:33.829233  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:34:33.941215  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 21:34:33.990376  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:34.002127  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:34.012178 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:34.020380  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:34.022949 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:34:34.037739  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:34:34.101827  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 21:34:34.105700 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:34.106479  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:34.175385 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:34.422920  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:34.424055  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:34.452811  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:34.453961 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:34.459496 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:34.460737 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:34.466703  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:34.514225  9173 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 67227.2us
W20240502 21:34:34.563386 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:34.724553  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 21:34:34.761317  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:34.766654 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:34.781791 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:34.783320 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:34.806946  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:34.813028  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:34.816164  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:34.941211 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:34.950263  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 21:34:35.144582 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:35.152693  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:35.153690  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:35.155589 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:35.155900  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:35.156842  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:35.233322 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:35.257203  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:34:35.370019 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:34:35.469612 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:35.501477  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:35.503979 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:35.516144 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:35.524097  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:35.528769  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:35.536603  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:35.580134 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:35.673585  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 21:34:35.838248  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:35.861359 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:34:35.886816  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 21:34:35.888250 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:35.888571  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 21:34:35.889237  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:35.889756  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:35.930644 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:35.989176 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:35.996358  9310 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.71:35857, user_credentials={real_user=slave}} blocked reactor thread for 76266.4us
I20240502 21:34:36.018074  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:36.206336 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:34:36.211040  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 21:34:36.221765  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:36.227454  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:36.230518  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:36.230715 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:36.233974 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:36.238204  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:36.309764  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:34:36.410810 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:36.477677  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 21:34:36.528244 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:36.532058  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:36.537811  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:36.552973  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:36.563491 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:36.564556 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:36.577566  9310 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.71:35857, user_credentials={real_user=slave}} blocked reactor thread for 47874.6us
I20240502 21:34:36.593998  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:36.625563  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 21:34:36.840034  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:36.850477 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:36.877240  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:36.891772 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:36.895071  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:36.954479 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:36.967355 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:36.987274  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:37.010916  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:34:37.012557  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 21:34:37.237264 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:37.237951  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:37.248137  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:37.252691 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:34:37.304595  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 21:34:37.307548  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:37.341902  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:37.342854 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:37.475872 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:37.481114  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 21:34:37.642336  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:37.673405 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:37.681397  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:37.682906 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:37.688169  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:34:37.689971 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 21:34:37.705658 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:37.722652  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:37.926143  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:34:38.029006 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:38.062907  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:38.071332 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:38.076187  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:38.098161  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:38.114785  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:38.116294 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:38.126200 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:38.174736  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:34:38.269009  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 21:34:38.445744  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:38.464071 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:38.464201 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:38.464707  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:38.473676  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:38.480234  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:38.482439  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 21:34:38.514328 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:38.531773 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:34:38.532894 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:38.776826  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 21:34:38.788935  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:38.793795  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:34:38.794204  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 21:34:38.802249  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:38.804917 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:38.825408 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:38.862571 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:38.868989  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 21:34:38.880347  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:38.968494  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:34:39.035815 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:39.113214  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 21:34:39.127676 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:39.130559  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:39.149622 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:39.157032  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:39.163219  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:39.165537 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:39.172624  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:39.323457  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 21:34:39.475896  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:39.498773  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:39.498961 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:39.503010  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:39.503131 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:39.512073  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:39.513832 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:39.535493 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:39.790542  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 21:34:39.854977 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:39.855756  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:39.876197  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:39.876997  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:39.907263 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:39.928253 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:39.931089 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:39.956496  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:40.039920 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 21:34:40.061169 20374 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:34:40.061365 20374 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:34:40.062114 20374 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:34:40.064435 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:40.064725 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
I20240502 21:34:40.064824 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:34:40.065454 20374 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 21:34:40.168242  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 21:34:40.254495  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:40.276247 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:40.283466  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:40.300284  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:40.308194 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:40.313267  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:34:40.331214 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:40.335673 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:40.356177  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:40.461668  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 21:34:40.643195  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:40.644196  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:40.645267  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:40.659425 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:34:40.660552  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 21:34:40.689877 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:40.718505 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:40.741083  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:40.836621  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:34:40.939976 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:40.981065 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:40.981428  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:40.981537  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:41.005229  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:41.013592 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:41.024884 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:41.039665  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:41.192363  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 21:34:41.310557  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:41.320911 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:41.324963  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:41.331578  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:41.354476 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:41.383025 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:41.393424  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:41.397750  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:34:41.406805  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:34:41.431243 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:41.501128  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 21:34:41.639513 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:41.644728  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:41.649148  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:41.681459  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:41.698937 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:41.720649  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:41.725731 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:41.742733  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:34:41.924968 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:41.951372  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 21:34:42.018250 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:42.023177  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:42.032438  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 21:34:42.044839  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:42.049470  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:42.064025  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:42.074934 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:42.090309 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:42.261209 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 21:34:42.382931  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:42.389206  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:42.398056 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:42.407439 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:42.411849 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:42.424707  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:42.433524 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:42.434190  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:42.676230  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:42.688066 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:42.704176  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:42.729856  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:42.737902  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:42.751468 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:42.757329 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:42.797164  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:34:42.904981 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:42.993450  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 21:34:43.045383  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:43.046121  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:43.048905 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:43.061107 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:43.073009  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:43.082973 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:43.086889  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:43.128221  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:34:43.203224  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:34:43.305735  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 21:34:43.453019  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:43.454860  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:43.460366  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:43.463433 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:43.467890 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:43.477586 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:43.479959  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:43.525983 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:43.630872  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:34:43.712142  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 21:34:43.818179  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:43.833949 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 22 similar messages]
I20240502 21:34:43.838007  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:43.860430 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:43.863428 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:43.884506  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:43.915082  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:43.921554 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:43.922256 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:43.966007  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:34:44.160434  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 21:34:44.204403  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:44.214784 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:44.227316 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:44.228328  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:44.233577  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:44.236742  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:44.261044  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:34:44.300706  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 21:34:44.301951 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:44.441243 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:44.550375 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:44.563093  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:44.568315  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:44.569523  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:44.569880 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:44.575968  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:44.591316 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:44.683207  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 21:34:44.856747 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:44.857971  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:44.864742  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:44.877398  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:44.881721 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:44.883149 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 21:34:44.883968 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:44.928952 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:44.968833  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:45.247383  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 21:34:45.271191 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:45.272907  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:45.297116  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:45.302485  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:45.331609 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:45.332300 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:45.333101  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240502 21:34:45.373220  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:45.417551 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:45.487037  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 21:34:45.590305  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:45.594326  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:45.594450 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:45.595587  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:45.629884 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:45.636251 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:45.643558  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 21:34:45.644850  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:45.831375  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 21:34:45.918038  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:45.928061  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:45.937769 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:45.940896  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:45.949123 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:45.964354  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:45.970078 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:45.999431 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:46.010399  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 21:34:46.235163  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:46.254832  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:46.256096  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:46.269904 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:46.279934 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:46.283823  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:46.290956 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:46.417138  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:34:46.422608  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 21:34:46.603341 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:46.604251  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:46.619374  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:46.620157  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:46.622893 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:46.627019 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:46.636525 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:46.653595  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:46.666029  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:34:46.697360  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 21:34:46.931808  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:46.947199  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:46.947898  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:46.949117 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:46.981062 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:47.009889 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:47.010303  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:47.026559  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:34:47.042627 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:47.074139  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240502 21:34:47.295135  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:47.295604 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:47.332233  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:47.332813  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:47.333878 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 21:34:47.339901 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:47.342113 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:47.407945  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:47.510030 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:47.678241 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:47.683740  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:47.698271  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:47.716179 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:47.717900  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 21:34:47.736011 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:47.758962  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:47.854166  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:47.970100 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:47.989241  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:34:48.016371  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 21:34:48.076711 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:48.081141  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:48.091146  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:48.097889  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:48.100438  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:48.114048 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:48.126415 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:48.360932  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:34:48.384968  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 21:34:48.461737  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:48.462455  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:48.462702 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:48.487604 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:48.492693  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:48.497836 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:48.530380 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:48.536314  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:48.731770  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 21:34:48.751945  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:48.765074 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:48.776154  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:48.806712  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:48.841925 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:48.859632  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 21:34:48.882987 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:48.883781  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:48.999554  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:34:49.042047 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:34:49.044051 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:49.104570  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:49.118165  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 21:34:49.138935  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:49.160831 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:49.182581 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:49.194324  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:49.194336 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:49.205339  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:49.350091  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:34:49.455149 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:49.464470  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:49.473459 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:49.485278  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:49.491003  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:49.516507 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:49.548169  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:49.556664  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 21:34:49.566339 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:49.697088  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:34:49.801343 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 21:34:49.867460  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:49.892568 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:49.896868  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:49.896872 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:49.900480 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:49.913657  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:49.947607  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:50.078802 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:50.182914  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:50.186178 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:50.189473  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:50.230664  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:50.244614 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:50.254534  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:50.294486 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:50.418505  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 21:34:50.522382 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:50.522425  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:50.536270  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:50.546590 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:50.549924 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:50.552318  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 21:34:50.566037  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:50.578325  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 21:34:50.584447  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:50.605248 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:50.642957  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:34:50.707681  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 21:34:50.851671  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:50.870554 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:50.874212  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:50.878165  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:50.891696  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:50.908142 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:50.910562 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:51.056819 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:51.191577  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:51.206737  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:51.228688  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:51.238328 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:51.249414 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:51.251916 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:51.278781  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:51.279184  9173 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 52645us
W20240502 21:34:51.303866  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:34:51.406503  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:34:51.436966  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 21:34:51.590058  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:51.623549  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:51.628651 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:51.629838  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:51.673457 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:51.681331 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:51.703251 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:51.723716  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 21:34:51.735488  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:51.735571 11028 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.66:39071, user_credentials={real_user=slave}} blocked reactor thread for 54035.3us
W20240502 21:34:51.971419  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 21:34:51.982232  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:51.999924  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:52.002120  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:52.007000 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:52.008710 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:52.026744  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 21:34:52.036316  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:52.036715 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:52.108120 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:34:52.160230 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:52.353731  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 21:34:52.374734  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:52.390936  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:52.403254 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:52.414613 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:52.417551  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:52.448793 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:52.472939  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:52.735010 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:52.740857  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:52.760351  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:52.763859  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:52.795864 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:52.799782  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:52.819264 11430 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 67896us
W20240502 21:34:52.822113 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:52.837010 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:52.947238  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:34:53.041664  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 21:34:53.106467  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:53.125973  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:53.133950 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:53.153890  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:53.166496 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:53.176338  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 21:34:53.184295 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:53.191293 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:53.200443  9447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.72:40467, user_credentials={real_user=slave}} blocked reactor thread for 65279.2us
I20240502 21:34:53.201151  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:53.240617  9039 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.69:42049, user_credentials={real_user=slave}} blocked reactor thread for 89143.7us
W20240502 21:34:53.347615  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 21:34:53.480393 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:53.481069  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:53.481137  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:53.514410 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:53.532244  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:53.566784 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:53.578194 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:53.597900  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:53.859941  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 21:34:53.866219  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:53.872005  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:53.872742  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:53.874042 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:53.878700 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:34:53.881965  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 21:34:53.894789 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:53.909183  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:53.995673  9175 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.72:40467, user_credentials={real_user=slave}} blocked reactor thread for 78079.3us
W20240502 21:34:54.056289 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:34:54.058001  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:34:54.058347 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:54.097910  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 21:34:54.185619  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:54.187350  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:54.190295  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:54.191308 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:54.215178 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:54.229714 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:54.237174  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:54.255486  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:34:54.473497  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 21:34:54.556411  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:54.566752  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:54.576831 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:54.582079 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:54.584944  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:54.585736 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:54.618141  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:54.637435  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:34:54.665756 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:34:54.665926 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:54.727974  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 21:34:54.821120 20374 raft_consensus.cc:482] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ef0beaf2f6941afb6ea9f9f274517fe)
I20240502 21:34:54.821339 20374 raft_consensus.cc:504] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ef0beaf2f6941afb6ea9f9f274517fe" member_type: VOTER last_known_addr { host: "127.27.196.70" port: 44211 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:34:54.822189 20374 leader_election.cc:288] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:34:54.824731 11430 leader_election.cc:341] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Not found: Tablet not found: f82c35077224414cb284472d49d7de4f
W20240502 21:34:54.825855 11430 leader_election.cc:334] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:54.826139 11430 leader_election.cc:302] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 6ef0beaf2f6941afb6ea9f9f274517fe, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:34:54.826471 20374 raft_consensus.cc:2726] T f82c35077224414cb284472d49d7de4f P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 21:34:54.923887  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:54.931879  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:54.935823 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:54.937834 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:54.939612  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:54.940557  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:54.942858 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:55.198096 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:55.295210 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:55.308887  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:55.316742  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:55.325387 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:55.333647  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:55.363677  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:55.392060 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:55.555857  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240502 21:34:55.571964  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:34:55.587847  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:34:55.616407 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:55.631531  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 21:34:55.704916  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:55.711853 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:55.712918  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:55.729398  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:55.799335 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:55.821290 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:55.847311  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:55.857409  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 21:34:56.087502  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:56.090062 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:56.117237 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:56.122143  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:56.135424 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:56.140074  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:56.162055 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:56.165868  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:56.304332  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 21:34:56.471199  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:56.471751 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:34:56.480598  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 21:34:56.492126  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:56.497736  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:56.498191 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:56.505250 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:56.506220  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:56.682109 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:56.690086  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:34:56.729326  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 21:34:56.786607  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:56.824753  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:56.825639 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:56.836673  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:56.839484 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:56.843293  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:56.881747 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:56.960508  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:34:56.968392  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:34:57.123806 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 21:34:57.166015  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:57.167894 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:57.171967  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:57.175624 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:57.186187  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:57.210242  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:57.228389 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:57.300192 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:57.491446  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 21:34:57.559967  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:57.580452  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:57.582329 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:57.596580 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:57.597864  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:57.610510 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:57.641031  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:57.740542  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:34:57.833141 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:57.879470  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:57.904512 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:57.910892  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:57.915421 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:57.943723  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:57.948254 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:57.959895  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:58.026469  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:34:58.078145  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 21:34:58.234501  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:58.252487 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:58.295557  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:34:58.308965  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 21:34:58.310216  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:58.344514 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:58.362990 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:58.375408  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:58.424103 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:58.650718  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:58.668994 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:58.669492  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:58.676291 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:58.709467  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:58.712754 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:58.756045  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:58.890131 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:34:58.940165  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:34:58.942209  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:34:58.990391  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 21:34:59.102912 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:59.106417  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:59.106911 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:59.110034  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:59.117863 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:59.147579  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:34:59.191630  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 21:34:59.275342  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:34:59.364272  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:34:59.452970  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 21:34:59.462877  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:59.473634 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:59.475051  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:59.477495 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:34:59.490463 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 21:34:59.491695 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:34:59.492723  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:59.496830 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:34:59.515069  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 21:34:59.515969  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:34:59.579849 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:34:59.674199  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 21:34:59.839188  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:34:59.839380 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:34:59.860471  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:34:59.897136 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:59.898691  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:34:59.916579 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:34:59.977671  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:35:00.089298 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:35:00.091205  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 21:35:00.218735  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:00.223354 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:35:00.237974  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:35:00.255132  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:00.255283  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:35:00.264686 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:00.282987 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:35:00.451462  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:35:00.471318  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:35:00.474087 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:35:00.536319 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:35:00.536909  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:35:00.542681  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:00.559933  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:00.560220 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:00.576617  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:35:00.603977 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:35:00.619854  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:35:00.639103  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 21:35:00.894464 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:35:00.901623  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:35:00.902349  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:00.914429  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:35:00.957893 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:00.967135 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:00.974403  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:35:01.055663 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:35:01.206702  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 21:35:01.239864 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:35:01.243955  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:01.265794  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:35:01.270751  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:35:01.310483 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:01.312223 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:01.336546  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:35:01.558099 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:35:01.580935  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:01.586977 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:35:01.598290  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:35:01.598857 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:01.610383 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:01.640933  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:01.643287  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:35:01.644661  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:35:01.713069  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:35:01.877146  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:35:01.935966  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 21:35:01.952899  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:35:01.967633 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:35:01.968640  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:02.029841  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:02.031466 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:02.031949 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:35:02.053442 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:35:02.131785  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:35:02.205708  9039 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:35:02.255556 11430 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 21:35:02.415616  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:35:02.439528  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:02.444664 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:35:02.449313 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:02.452553 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:02.466964  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:35:02.548784  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:35:02.566233 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:35:02.723232  9445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 21:35:02.752202  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:35:02.759399  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 21:35:02.770292 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:02.771178  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:35:02.798218 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:02.801589  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:02.832554  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:35:02.887686 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:35:02.933957  9311 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:35:03.007001 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:35:03.045079  9311 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 21:35:03.121510  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:35:03.132591 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:35:03.136266  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:35:03.141891  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:03.186025 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:03.205641 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:03.254153  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:35:03.281139  9173 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.70:44211, user_credentials={real_user=slave}} blocked reactor thread for 55439.5us
W20240502 21:35:03.360669  9445 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 21:35:03.452431 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:35:03.491888 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:35:03.491961  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:03.511632  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:35:03.523909  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:35:03.526497  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:03.527226 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:03.543627 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:35:03.625255  9173 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 21:35:03.829643  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:03.853927 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:35:03.857976  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:35:03.869565  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:35:03.885433  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:03.891717 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:03.902737 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:35:03.982754 11430 consensus_peers.cc:462] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:35:04.109036  9039 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:35:04.138417  9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 21:35:04.210708  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:35:04.230273  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:04.233022 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:04.238065  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:35:04.244022 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:35:04.287782 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:04.296849  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:35:04.326010  9175 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.72:40467, user_credentials={real_user=slave}} blocked reactor thread for 43649.2us
W20240502 21:35:04.342491  9445 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 21:35:04.364746  9445 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 21:35:04.445523 24497 raft_consensus.cc:998] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2: Attempting to remove follower 6ef0beaf2f6941afb6ea9f9f274517fe from the Raft config. Reason: Leader has been unable to successfully communicate with peer 6ef0beaf2f6941afb6ea9f9f274517fe for more than 300 seconds (300.338s)
I20240502 21:35:04.447072 24497 consensus_queue.cc:237] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:04.449580 11093 raft_consensus.cc:1261] T a7c6b881c9ef4cdca787f065778e526f P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Refusing update from remote peer e0de0545fa0547d3929111e990e6d1b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 21:35:04.450309 24497 consensus_queue.cc:1035] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:35:04.454247 20374 raft_consensus.cc:2932] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:35:04.460011 11093 raft_consensus.cc:2932] T a7c6b881c9ef4cdca787f065778e526f P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:35:04.462260  8958 catalog_manager.cc:5484] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 reported cstate change: config changed from index 10 to 11, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New cstate: current_term: 2 leader_uuid: "e0de0545fa0547d3929111e990e6d1b2" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:35:04.509313 20374 raft_consensus.cc:998] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2: Attempting to remove follower 6ef0beaf2f6941afb6ea9f9f274517fe from the Raft config. Reason: Leader has been unable to successfully communicate with peer 6ef0beaf2f6941afb6ea9f9f274517fe for more than 300 seconds (300.062s)
I20240502 21:35:04.510308 20374 consensus_queue.cc:237] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
W20240502 21:35:04.513610 11430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 21:35:04.516341 23741 raft_consensus.cc:998] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670: Attempting to remove follower 6ef0beaf2f6941afb6ea9f9f274517fe from the Raft config. Reason: Leader has been unable to successfully communicate with peer 6ef0beaf2f6941afb6ea9f9f274517fe for more than 300 seconds (300.095s)
I20240502 21:35:04.517680 23741 consensus_queue.cc:237] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
W20240502 21:35:04.521221  8946 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:35:04.532958  8946 catalog_manager.cc:4573] TS 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a7c6b881c9ef4cdca787f065778e526f: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:35:04.533624  9107 raft_consensus.cc:1261] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Refusing update from remote peer e0de0545fa0547d3929111e990e6d1b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:35:04.534443 20374 consensus_queue.cc:1035] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:35:04.535282 11093 raft_consensus.cc:1261] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:35:04.535315  9377 raft_consensus.cc:1261] T 3caa36212ba444aa88d153ff9f90122e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:35:04.535959 24478 consensus_queue.cc:1035] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:35:04.537441 23741 consensus_queue.cc:1035] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:35:04.538547 20374 raft_consensus.cc:2932] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:35:04.540189  9107 raft_consensus.cc:2932] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:35:04.542140 24478 raft_consensus.cc:2932] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:35:04.548637  9377 raft_consensus.cc:2932] T 3caa36212ba444aa88d153ff9f90122e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
W20240502 21:35:04.559548  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 21:35:04.561419  8959 catalog_manager.cc:5484] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: config changed from index 6 to 7, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:35:04.569734  8953 catalog_manager.cc:5484] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 reported cstate change: config changed from index 8 to 9, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New cstate: current_term: 2 leader_uuid: "e0de0545fa0547d3929111e990e6d1b2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:35:04.570252 24481 raft_consensus.cc:998] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0: Attempting to remove follower 6ef0beaf2f6941afb6ea9f9f274517fe from the Raft config. Reason: Leader has been unable to successfully communicate with peer 6ef0beaf2f6941afb6ea9f9f274517fe for more than 300 seconds (300.032s)
I20240502 21:35:04.571156 24481 consensus_queue.cc:237] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:35:04.583000 11093 raft_consensus.cc:2932] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:35:04.594719 11496 raft_consensus.cc:1261] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:35:04.596055 24424 consensus_queue.cc:1035] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:35:04.607028  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:04.620227 24424 raft_consensus.cc:2932] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:35:04.621804 24526 raft_consensus.cc:998] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58: Attempting to remove follower 6ef0beaf2f6941afb6ea9f9f274517fe from the Raft config. Reason: Leader has been unable to successfully communicate with peer 6ef0beaf2f6941afb6ea9f9f274517fe for more than 300 seconds (300.271s)
I20240502 21:35:04.622577 24526 consensus_queue.cc:237] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:35:04.635372 11494 consensus_queue.cc:237] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:35:04.637485  8954 catalog_manager.cc:5484] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: config changed from index 7 to 8, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:35:04.641340 11494 consensus_queue.cc:237] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:35:04.644286  9107 raft_consensus.cc:1261] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Refusing update from remote peer e0de0545fa0547d3929111e990e6d1b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 21:35:04.645651 24497 consensus_queue.cc:1035] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:35:04.645961 11093 raft_consensus.cc:1261] T a7c6b881c9ef4cdca787f065778e526f P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Refusing update from remote peer e0de0545fa0547d3929111e990e6d1b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 21:35:04.646700 24497 consensus_queue.cc:1035] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:35:04.649894 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:35:04.650605 11431 consensus_peers.cc:485] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Couldn't send request to peer f2fdd596a8b44bbc848fe40f9f07fbe0. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:04.650967 11496 raft_consensus.cc:2932] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } }
I20240502 21:35:04.658178 11365 raft_consensus.cc:1261] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:35:04.658008 24497 raft_consensus.cc:2932] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } }
I20240502 21:35:04.659016 24436 consensus_queue.cc:1035] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 21:35:04.662074 11432 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Couldn't send request to peer 525a093ea5bf40dc9bbf9dc39895d670. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:04.665545  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:35:04.664752 11093 raft_consensus.cc:2932] T a7c6b881c9ef4cdca787f065778e526f P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } }
I20240502 21:35:04.690351  9377 consensus_queue.cc:237] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:35:04.690939  8954 catalog_manager.cc:5484] T a7c6b881c9ef4cdca787f065778e526f P d10e9e156ba74a97a221c85b68ab8bc4 reported cstate change: config changed from index 11 to 12, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) added. New cstate: current_term: 2 leader_uuid: "e0de0545fa0547d3929111e990e6d1b2" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } }
I20240502 21:35:04.699573  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:35:04.707645  9310 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Couldn't send request to peer 5543ddab5b484642b139b7c848932a58. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:04.733742 24539 raft_consensus.cc:2932] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } }
W20240502 21:35:04.740458 11430 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.69:42049, user_credentials={real_user=slave}} blocked reactor thread for 54662.3us
I20240502 21:35:04.741813 24219 raft_consensus.cc:998] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6: Attempting to remove follower 6ef0beaf2f6941afb6ea9f9f274517fe from the Raft config. Reason: Leader has been unable to successfully communicate with peer 6ef0beaf2f6941afb6ea9f9f274517fe for more than 300 seconds (300.236s)
I20240502 21:35:04.742280 24526 raft_consensus.cc:998] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58: Attempting to remove follower 6ef0beaf2f6941afb6ea9f9f274517fe from the Raft config. Reason: Leader has been unable to successfully communicate with peer 6ef0beaf2f6941afb6ea9f9f274517fe for more than 300 seconds (300.224s)
I20240502 21:35:04.743228 24526 consensus_queue.cc:237] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:35:04.742877 24219 consensus_queue.cc:237] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:35:04.752281 11365 raft_consensus.cc:2932] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:35:04.755448 24436 raft_consensus.cc:2932] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:35:04.758952 11494 raft_consensus.cc:1261] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:35:04.763871 24424 consensus_queue.cc:1035] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:35:04.767839 24481 raft_consensus.cc:2932] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } }
I20240502 21:35:04.768620  9107 raft_consensus.cc:2932] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } }
I20240502 21:35:04.768618 11494 raft_consensus.cc:2932] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } }
I20240502 21:35:04.774109  8948 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 4674e376bf7440cda619784491685b44 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 21:35:04.783003  8954 catalog_manager.cc:5484] T 94d2cdf8ae6941668b5f680c558656e8 P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index 9 to 10, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) added. New cstate: current_term: 2 leader_uuid: "e0de0545fa0547d3929111e990e6d1b2" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } }
I20240502 21:35:04.785048  8953 catalog_manager.cc:5484] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 reported cstate change: config changed from index 7 to 8, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:35:04.787405  8949 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet a7c6b881c9ef4cdca787f065778e526f with cas_config_opid_index 11: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 21:35:04.796443  8959 catalog_manager.cc:5484] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2 reported cstate change: config changed from index 8 to 9, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) added. New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } }
I20240502 21:35:04.803552 24551 raft_consensus.cc:998] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6: Attempting to remove follower 6ef0beaf2f6941afb6ea9f9f274517fe from the Raft config. Reason: Leader has been unable to successfully communicate with peer 6ef0beaf2f6941afb6ea9f9f274517fe for more than 300 seconds (300.202s)
I20240502 21:35:04.804904 24551 consensus_queue.cc:237] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:04.815765  8949 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 94d2cdf8ae6941668b5f680c558656e8 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 21:35:04.835090  9507 raft_consensus.cc:1261] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:35:04.838061 24526 consensus_queue.cc:1035] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:35:04.850056 24552 raft_consensus.cc:998] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6: Attempting to remove follower 6ef0beaf2f6941afb6ea9f9f274517fe from the Raft config. Reason: Leader has been unable to successfully communicate with peer 6ef0beaf2f6941afb6ea9f9f274517fe for more than 300 seconds (300.259s)
I20240502 21:35:04.850726 11093 raft_consensus.cc:1261] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:35:04.852463 24551 consensus_queue.cc:1035] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:35:04.856642  9107 consensus_queue.cc:237] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:35:04.850935 24552 consensus_queue.cc:237] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:35:04.860257 11365 raft_consensus.cc:1261] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:35:04.860756 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:04.861182 24436 consensus_queue.cc:1035] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:35:04.866472 24544 raft_consensus.cc:2932] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } }
I20240502 21:35:04.867157 11365 raft_consensus.cc:2932] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } }
W20240502 21:35:04.874720  9038 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58 -> Peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): Couldn't send request to peer 19f162e164854f2da623a4bb8f9e6bf6. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:04.875572  8959 catalog_manager.cc:5484] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 reported cstate change: config changed from index 8 to 9, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) added. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } }
I20240502 21:35:04.884716  9239 raft_consensus.cc:1261] T c56ac926025e47218e9df4088184a278 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:35:04.885412 24480 consensus_queue.cc:1035] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:35:04.889160 24551 raft_consensus.cc:2932] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:35:04.890733  8945 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 32c306d724954a16be21a34aa4cc0b5f with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 21:35:04.898833 24551 raft_consensus.cc:2932] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } }
I20240502 21:35:04.903342 11093 raft_consensus.cc:2932] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:35:04.904165  9239 raft_consensus.cc:2932] T c56ac926025e47218e9df4088184a278 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } }
I20240502 21:35:04.905352  9107 raft_consensus.cc:1261] T 3d1a5050d3f944969a560994a23700ad P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:35:04.909478 24480 consensus_queue.cc:1035] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:35:04.911154  8959 catalog_manager.cc:5484] T c56ac926025e47218e9df4088184a278 P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: config changed from index 5 to 6, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New cstate: current_term: 1 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } }
I20240502 21:35:04.916277  8953 catalog_manager.cc:5484] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: config changed from index 8 to 9, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New cstate: current_term: 2 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:35:04.920255  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:04.938553 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:04.940554  9508 consensus_queue.cc:237] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:35:04.942047 24544 raft_consensus.cc:2932] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } }
W20240502 21:35:04.943145  9444 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Couldn't send request to peer 5543ddab5b484642b139b7c848932a58. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:04.944085  9508 consensus_queue.cc:237] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:35:04.947058  9507 raft_consensus.cc:2932] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } }
W20240502 21:35:04.952050  9444 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Couldn't send request to peer 5543ddab5b484642b139b7c848932a58. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:04.956209  8953 catalog_manager.cc:5484] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: config changed from index 5 to 6, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } }
I20240502 21:35:04.956570 11093 raft_consensus.cc:1261] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 21:35:04.957692 24552 consensus_queue.cc:1035] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:35:04.963579  9239 raft_consensus.cc:1261] T c56ac926025e47218e9df4088184a278 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:35:04.964859 24566 consensus_queue.cc:1035] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:35:04.966962  9105 consensus_queue.cc:237] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
W20240502 21:35:04.969732  9040 consensus_peers.cc:485] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 -> Peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Couldn't send request to peer f2fdd596a8b44bbc848fe40f9f07fbe0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc1970b9263d41abb575b9d8639c59ae. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:04.970347  9508 raft_consensus.cc:1261] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:35:04.971503 24436 consensus_queue.cc:1035] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:35:04.977571 24551 raft_consensus.cc:2932] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } }
I20240502 21:35:04.982374  9107 raft_consensus.cc:2932] T 3d1a5050d3f944969a560994a23700ad P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } }
I20240502 21:35:04.984040 24549 raft_consensus.cc:998] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0: Attempting to remove follower 6ef0beaf2f6941afb6ea9f9f274517fe from the Raft config. Reason: Leader has been unable to successfully communicate with peer 6ef0beaf2f6941afb6ea9f9f274517fe for more than 300 seconds (300.582s)
I20240502 21:35:04.984740 24549 consensus_queue.cc:237] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
W20240502 21:35:04.986588  9311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 21:35:04.987594 24480 raft_consensus.cc:2932] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } }
I20240502 21:35:04.986761 11093 raft_consensus.cc:2932] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } }
I20240502 21:35:04.991519  8959 catalog_manager.cc:5484] T 3d1a5050d3f944969a560994a23700ad P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index 5 to 6, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New cstate: current_term: 1 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } }
I20240502 21:35:04.992159 24436 raft_consensus.cc:2932] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } }
I20240502 21:35:04.993250 24552 raft_consensus.cc:2932] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } }
I20240502 21:35:04.999329  8945 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet c56ac926025e47218e9df4088184a278 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 21:35:05.002424  8953 catalog_manager.cc:5484] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: config changed from index 6 to 7, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) added. New cstate: current_term: 1 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } }
I20240502 21:35:04.998667  9239 raft_consensus.cc:2932] T c56ac926025e47218e9df4088184a278 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } }
I20240502 21:35:05.002671  9507 raft_consensus.cc:2932] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } }
I20240502 21:35:05.004094  8945 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet dc1970b9263d41abb575b9d8639c59ae with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 21:35:05.005388  8954 catalog_manager.cc:5484] T be34ec89876d42b69efe1878c0d9bc19 P d10e9e156ba74a97a221c85b68ab8bc4 reported cstate change: config changed from index 9 to 10, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) added. New cstate: current_term: 2 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } }
I20240502 21:35:05.007809 24526 raft_consensus.cc:998] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58: Attempting to remove follower 6ef0beaf2f6941afb6ea9f9f274517fe from the Raft config. Reason: Leader has been unable to successfully communicate with peer 6ef0beaf2f6941afb6ea9f9f274517fe for more than 300 seconds (300.435s)
I20240502 21:35:05.008523 24526 consensus_queue.cc:237] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:35:05.020124  8953 catalog_manager.cc:5484] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: config changed from index 6 to 7, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) added. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } }
I20240502 21:35:05.026100  9507 raft_consensus.cc:1261] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:35:05.036964 24526 consensus_queue.cc:1035] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20240502 21:35:05.039772  8945 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet be34ec89876d42b69efe1878c0d9bc19 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 21:35:05.039577  9507 consensus_queue.cc:237] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:35:05.044713 11093 raft_consensus.cc:1261] T 8a2e06253c51405fa9b6bc9e87844760 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:35:05.046051 24549 consensus_queue.cc:1035] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 21:35:05.051592  9446 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Couldn't send request to peer f2fdd596a8b44bbc848fe40f9f07fbe0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d1a5050d3f944969a560994a23700ad. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:05.052784  9107 raft_consensus.cc:1261] T 3d1a5050d3f944969a560994a23700ad P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Refusing update from remote peer 19f162e164854f2da623a4bb8f9e6bf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:35:05.053642 24551 consensus_queue.cc:1035] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:35:05.057476 24480 raft_consensus.cc:2932] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } }
I20240502 21:35:05.061506 24574 ts_tablet_manager.cc:902] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0: Initiating tablet copy from peer e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467)
I20240502 21:35:05.063167  8945 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 3d1a5050d3f944969a560994a23700ad with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 21:35:05.065675  9107 raft_consensus.cc:2932] T 3d1a5050d3f944969a560994a23700ad P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } }
I20240502 21:35:05.066849  8953 catalog_manager.cc:5484] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 reported cstate change: config changed from index 6 to 7, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) added. New cstate: current_term: 1 leader_uuid: "19f162e164854f2da623a4bb8f9e6bf6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } }
I20240502 21:35:05.079488 24424 raft_consensus.cc:2932] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:35:05.082870 24436 raft_consensus.cc:2932] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } }
I20240502 21:35:05.083671  9508 raft_consensus.cc:2932] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } }
I20240502 21:35:05.087862 11094 raft_consensus.cc:2932] T 8a2e06253c51405fa9b6bc9e87844760 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:35:05.095940  8955 catalog_manager.cc:5484] T 8a2e06253c51405fa9b6bc9e87844760 P d10e9e156ba74a97a221c85b68ab8bc4 reported cstate change: config changed from index 7 to 8, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:35:05.103519  9377 consensus_queue.cc:237] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:35:05.104833  8959 catalog_manager.cc:5484] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index 5 to 6, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } }
W20240502 21:35:05.108579  9313 consensus_peers.cc:485] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Couldn't send request to peer 525a093ea5bf40dc9bbf9dc39895d670. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:05.108827 11093 raft_consensus.cc:1261] T 8a2e06253c51405fa9b6bc9e87844760 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Refusing update from remote peer f2fdd596a8b44bbc848fe40f9f07fbe0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:35:05.109539 24424 consensus_queue.cc:1035] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:35:05.112808 24424 raft_consensus.cc:2932] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } }
I20240502 21:35:05.117341 11093 raft_consensus.cc:2932] T 8a2e06253c51405fa9b6bc9e87844760 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } }
I20240502 21:35:05.119889  9107 consensus_queue.cc:237] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:35:05.124250  9508 raft_consensus.cc:1261] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 21:35:05.125299  9038 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857): Couldn't send request to peer 4cbb5b06a6a44eb6b2145cd538b01af2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9240163cfac446d2a14dbd9fcbbf9ff7. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:05.125540 24526 consensus_queue.cc:1035] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:35:05.127071  8948 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 8a2e06253c51405fa9b6bc9e87844760 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 21:35:05.128916 24526 raft_consensus.cc:2932] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:35:05.130865  8959 catalog_manager.cc:5484] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0 reported cstate change: config changed from index 8 to 9, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) added. New cstate: current_term: 1 leader_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } }
I20240502 21:35:05.132057  9508 raft_consensus.cc:2932] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:35:05.141295  8945 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 9240163cfac446d2a14dbd9fcbbf9ff7 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 21:35:05.149655  8953 catalog_manager.cc:5484] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index 6 to 7, VOTER 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71) added. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:35:05.160950 24574 tablet_copy_client.cc:308] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.72:40467
I20240502 21:35:05.163729 11517 tablet_copy_service.cc:140] P e0de0545fa0547d3929111e990e6d1b2: Received BeginTabletCopySession request for tablet a7c6b881c9ef4cdca787f065778e526f from peer f2fdd596a8b44bbc848fe40f9f07fbe0 ({username='slave'} at 127.27.196.67:39199)
I20240502 21:35:05.163924 11517 tablet_copy_service.cc:161] P e0de0545fa0547d3929111e990e6d1b2: Beginning new tablet copy session on tablet a7c6b881c9ef4cdca787f065778e526f from peer f2fdd596a8b44bbc848fe40f9f07fbe0 at {username='slave'} at 127.27.196.67:39199: session id = f2fdd596a8b44bbc848fe40f9f07fbe0-a7c6b881c9ef4cdca787f065778e526f
I20240502 21:35:05.168568 11517 tablet_copy_source_session.cc:205] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:35:05.171985 24574 ts_tablet_manager.cc:1875] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:35:05.182058 24574 ts_tablet_manager.cc:1888] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240502 21:35:05.186955 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7c6b881c9ef4cdca787f065778e526f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:05.191995 24574 tablet_copy_client.cc:783] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0: tablet copy: Starting download of 5 data blocks...
I20240502 21:35:05.204798 24574 tablet_copy_client.cc:647] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:35:05.207844 24574 tablet_copy_client.cc:523] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:35:05.217825 24574 tablet_bootstrap.cc:492] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap starting.
I20240502 21:35:05.219872 24583 ts_tablet_manager.cc:902] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670: Initiating tablet copy from peer e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467)
I20240502 21:35:05.291414 24583 tablet_copy_client.cc:308] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.72:40467
I20240502 21:35:05.292526 11517 tablet_copy_service.cc:140] P e0de0545fa0547d3929111e990e6d1b2: Received BeginTabletCopySession request for tablet 94d2cdf8ae6941668b5f680c558656e8 from peer 525a093ea5bf40dc9bbf9dc39895d670 ({username='slave'} at 127.27.196.66:44945)
I20240502 21:35:05.292738 11517 tablet_copy_service.cc:161] P e0de0545fa0547d3929111e990e6d1b2: Beginning new tablet copy session on tablet 94d2cdf8ae6941668b5f680c558656e8 from peer 525a093ea5bf40dc9bbf9dc39895d670 at {username='slave'} at 127.27.196.66:44945: session id = 525a093ea5bf40dc9bbf9dc39895d670-94d2cdf8ae6941668b5f680c558656e8
I20240502 21:35:05.296770 11517 tablet_copy_source_session.cc:205] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:35:05.299727 24583 ts_tablet_manager.cc:1875] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:35:05.303345 24589 ts_tablet_manager.cc:902] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58: Initiating tablet copy from peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:35:05.350437 24583 ts_tablet_manager.cc:1888] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240502 21:35:05.350925 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94d2cdf8ae6941668b5f680c558656e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:05.354422 24593 ts_tablet_manager.cc:902] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6: Initiating tablet copy from peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:35:05.364049 24592 ts_tablet_manager.cc:902] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58: Initiating tablet copy from peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
I20240502 21:35:05.367250 24583 tablet_copy_client.cc:783] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670: tablet copy: Starting download of 5 data blocks...
I20240502 21:35:05.378050 24574 tablet_bootstrap.cc:492] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:35:05.378091 24583 tablet_copy_client.cc:647] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670: tablet copy: Starting download of 1 WAL segments...
I20240502 21:35:05.378692 24574 tablet_bootstrap.cc:492] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap complete.
I20240502 21:35:05.379086 24574 ts_tablet_manager.cc:1367] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent bootstrapping tablet: real 0.162s	user 0.059s	sys 0.035s
I20240502 21:35:05.380170 24583 tablet_copy_client.cc:523] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:35:05.385828 24574 raft_consensus.cc:348] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:35:05.386126 24574 raft_consensus.cc:729] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Initialized, Role: FOLLOWER
I20240502 21:35:05.387576 24574 consensus_queue.cc:260] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:35:05.391649 24574 ts_tablet_manager.cc:1398] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent starting tablet: real 0.012s	user 0.000s	sys 0.002s
I20240502 21:35:05.392495 11517 tablet_copy_service.cc:342] P e0de0545fa0547d3929111e990e6d1b2: Request end of tablet copy session f2fdd596a8b44bbc848fe40f9f07fbe0-a7c6b881c9ef4cdca787f065778e526f received from {username='slave'} at 127.27.196.67:39199
I20240502 21:35:05.392627 11517 tablet_copy_service.cc:433] P e0de0545fa0547d3929111e990e6d1b2: ending tablet copy session f2fdd596a8b44bbc848fe40f9f07fbe0-a7c6b881c9ef4cdca787f065778e526f on tablet a7c6b881c9ef4cdca787f065778e526f with peer f2fdd596a8b44bbc848fe40f9f07fbe0
I20240502 21:35:05.408166 24592 tablet_copy_client.cc:308] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.68:40931
I20240502 21:35:05.409287  9533 tablet_copy_service.cc:140] P 19f162e164854f2da623a4bb8f9e6bf6: Received BeginTabletCopySession request for tablet be34ec89876d42b69efe1878c0d9bc19 from peer 5543ddab5b484642b139b7c848932a58 ({username='slave'} at 127.27.196.65:48061)
I20240502 21:35:05.409579  9533 tablet_copy_service.cc:161] P 19f162e164854f2da623a4bb8f9e6bf6: Beginning new tablet copy session on tablet be34ec89876d42b69efe1878c0d9bc19 from peer 5543ddab5b484642b139b7c848932a58 at {username='slave'} at 127.27.196.65:48061: session id = 5543ddab5b484642b139b7c848932a58-be34ec89876d42b69efe1878c0d9bc19
I20240502 21:35:05.419310  9533 tablet_copy_source_session.cc:205] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:35:05.421947 24592 ts_tablet_manager.cc:1875] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:35:05.423751 24583 tablet_bootstrap.cc:492] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap starting.
I20240502 21:35:05.424551 24589 tablet_copy_client.cc:308] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:43203
I20240502 21:35:05.425551  9399 tablet_copy_service.cc:140] P f2fdd596a8b44bbc848fe40f9f07fbe0: Received BeginTabletCopySession request for tablet 4674e376bf7440cda619784491685b44 from peer 5543ddab5b484642b139b7c848932a58 ({username='slave'} at 127.27.196.65:38501)
I20240502 21:35:05.425745  9399 tablet_copy_service.cc:161] P f2fdd596a8b44bbc848fe40f9f07fbe0: Beginning new tablet copy session on tablet 4674e376bf7440cda619784491685b44 from peer 5543ddab5b484642b139b7c848932a58 at {username='slave'} at 127.27.196.65:38501: session id = 5543ddab5b484642b139b7c848932a58-4674e376bf7440cda619784491685b44
I20240502 21:35:05.429858  9399 tablet_copy_source_session.cc:205] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:35:05.433946 24589 ts_tablet_manager.cc:1875] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:35:05.434913 24592 ts_tablet_manager.cc:1888] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240502 21:35:05.435246 24592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be34ec89876d42b69efe1878c0d9bc19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:05.442834  9426 maintenance_manager.cc:419] P f2fdd596a8b44bbc848fe40f9f07fbe0: Scheduling UndoDeltaBlockGCOp(a7c6b881c9ef4cdca787f065778e526f): 4925 bytes on disk
I20240502 21:35:05.449383  9314 maintenance_manager.cc:643] P f2fdd596a8b44bbc848fe40f9f07fbe0: UndoDeltaBlockGCOp(a7c6b881c9ef4cdca787f065778e526f) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20240502 21:35:05.451877 24593 tablet_copy_client.cc:308] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:42177
I20240502 21:35:05.452661 24574 ts_tablet_manager.cc:902] T 3d1a5050d3f944969a560994a23700ad P f2fdd596a8b44bbc848fe40f9f07fbe0: Initiating tablet copy from peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
I20240502 21:35:05.453207  9128 tablet_copy_service.cc:140] P 5543ddab5b484642b139b7c848932a58: Received BeginTabletCopySession request for tablet 32c306d724954a16be21a34aa4cc0b5f from peer 19f162e164854f2da623a4bb8f9e6bf6 ({username='slave'} at 127.27.196.68:53551)
I20240502 21:35:05.453382  9128 tablet_copy_service.cc:161] P 5543ddab5b484642b139b7c848932a58: Beginning new tablet copy session on tablet 32c306d724954a16be21a34aa4cc0b5f from peer 19f162e164854f2da623a4bb8f9e6bf6 at {username='slave'} at 127.27.196.68:53551: session id = 19f162e164854f2da623a4bb8f9e6bf6-32c306d724954a16be21a34aa4cc0b5f
I20240502 21:35:05.458465  9128 tablet_copy_source_session.cc:205] T 32c306d724954a16be21a34aa4cc0b5f P 5543ddab5b484642b139b7c848932a58: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:35:05.459774 24592 tablet_copy_client.cc:783] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58: tablet copy: Starting download of 5 data blocks...
I20240502 21:35:05.470315 24589 ts_tablet_manager.cc:1888] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 21:35:05.470885 24589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4674e376bf7440cda619784491685b44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:05.476027 24583 tablet_bootstrap.cc:492] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:35:05.476778 24583 tablet_bootstrap.cc:492] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap complete.
I20240502 21:35:05.477072 24589 tablet_copy_client.cc:783] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58: tablet copy: Starting download of 5 data blocks...
I20240502 21:35:05.477169 24583 ts_tablet_manager.cc:1367] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670: Time spent bootstrapping tablet: real 0.054s	user 0.032s	sys 0.003s
I20240502 21:35:05.478255 24583 raft_consensus.cc:348] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:35:05.478490 24583 raft_consensus.cc:729] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Initialized, Role: FOLLOWER
I20240502 21:35:05.478837 24583 consensus_queue.cc:260] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:35:05.479815 24583 ts_tablet_manager.cc:1398] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 21:35:05.480792 11517 tablet_copy_service.cc:342] P e0de0545fa0547d3929111e990e6d1b2: Request end of tablet copy session 525a093ea5bf40dc9bbf9dc39895d670-94d2cdf8ae6941668b5f680c558656e8 received from {username='slave'} at 127.27.196.66:44945
I20240502 21:35:05.480918 11517 tablet_copy_service.cc:433] P e0de0545fa0547d3929111e990e6d1b2: ending tablet copy session 525a093ea5bf40dc9bbf9dc39895d670-94d2cdf8ae6941668b5f680c558656e8 on tablet 94d2cdf8ae6941668b5f680c558656e8 with peer 525a093ea5bf40dc9bbf9dc39895d670
I20240502 21:35:05.484262 24574 tablet_copy_client.cc:308] T 3d1a5050d3f944969a560994a23700ad P f2fdd596a8b44bbc848fe40f9f07fbe0: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.68:40931
I20240502 21:35:05.486526 24593 ts_tablet_manager.cc:1875] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:35:05.491741 24592 tablet_copy_client.cc:647] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58: tablet copy: Starting download of 1 WAL segments...
I20240502 21:35:05.495165 24583 ts_tablet_manager.cc:902] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670: Initiating tablet copy from peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:35:05.502552 24592 tablet_copy_client.cc:523] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:35:05.503676 24593 ts_tablet_manager.cc:1888] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 21:35:05.504091 24593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32c306d724954a16be21a34aa4cc0b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:05.512842 24611 ts_tablet_manager.cc:902] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58: Initiating tablet copy from peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
I20240502 21:35:05.516024 24592 tablet_bootstrap.cc:492] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58: Bootstrap starting.
I20240502 21:35:05.520316 24593 tablet_copy_client.cc:783] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6: tablet copy: Starting download of 5 data blocks...
I20240502 21:35:05.529381 24622 ts_tablet_manager.cc:902] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2: Initiating tablet copy from peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:35:05.535257 24593 tablet_copy_client.cc:647] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6: tablet copy: Starting download of 1 WAL segments...
I20240502 21:35:05.536458 24589 tablet_copy_client.cc:647] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58: tablet copy: Starting download of 1 WAL segments...
I20240502 21:35:05.538301 24593 tablet_copy_client.cc:523] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:35:05.539369 24589 tablet_copy_client.cc:523] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:35:05.546909 24589 tablet_bootstrap.cc:492] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58: Bootstrap starting.
I20240502 21:35:05.555974 24627 ts_tablet_manager.cc:902] T dc1970b9263d41abb575b9d8639c59ae P f2fdd596a8b44bbc848fe40f9f07fbe0: Initiating tablet copy from peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:35:05.556427 24593 tablet_bootstrap.cc:492] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6: Bootstrap starting.
W20240502 21:35:05.558207  8946 catalog_manager.cc:4573] TS 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 32c306d724954a16be21a34aa4cc0b5f: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:35:05.563380 24627 tablet_copy_client.cc:308] T dc1970b9263d41abb575b9d8639c59ae P f2fdd596a8b44bbc848fe40f9f07fbe0: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:42177
I20240502 21:35:05.568302  9377 raft_consensus.cc:1203] T a7c6b881c9ef4cdca787f065778e526f P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12]   Dedup: 2.12->[]
I20240502 21:35:05.569067  9128 tablet_copy_service.cc:140] P 5543ddab5b484642b139b7c848932a58: Received BeginTabletCopySession request for tablet dc1970b9263d41abb575b9d8639c59ae from peer f2fdd596a8b44bbc848fe40f9f07fbe0 ({username='slave'} at 127.27.196.67:40917)
I20240502 21:35:05.569274  9128 tablet_copy_service.cc:161] P 5543ddab5b484642b139b7c848932a58: Beginning new tablet copy session on tablet dc1970b9263d41abb575b9d8639c59ae from peer f2fdd596a8b44bbc848fe40f9f07fbe0 at {username='slave'} at 127.27.196.67:40917: session id = f2fdd596a8b44bbc848fe40f9f07fbe0-dc1970b9263d41abb575b9d8639c59ae
I20240502 21:35:05.576418  9128 tablet_copy_source_session.cc:205] T dc1970b9263d41abb575b9d8639c59ae P 5543ddab5b484642b139b7c848932a58: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:35:05.578413 24611 tablet_copy_client.cc:308] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.68:40931
I20240502 21:35:05.579468 24627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc1970b9263d41abb575b9d8639c59ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:05.580191  9533 tablet_copy_service.cc:140] P 19f162e164854f2da623a4bb8f9e6bf6: Received BeginTabletCopySession request for tablet c56ac926025e47218e9df4088184a278 from peer 5543ddab5b484642b139b7c848932a58 ({username='slave'} at 127.27.196.65:48061)
I20240502 21:35:05.580399  9533 tablet_copy_service.cc:161] P 19f162e164854f2da623a4bb8f9e6bf6: Beginning new tablet copy session on tablet c56ac926025e47218e9df4088184a278 from peer 5543ddab5b484642b139b7c848932a58 at {username='slave'} at 127.27.196.65:48061: session id = 5543ddab5b484642b139b7c848932a58-c56ac926025e47218e9df4088184a278
I20240502 21:35:05.582209  9532 tablet_copy_service.cc:140] P 19f162e164854f2da623a4bb8f9e6bf6: Received BeginTabletCopySession request for tablet 3d1a5050d3f944969a560994a23700ad from peer f2fdd596a8b44bbc848fe40f9f07fbe0 ({username='slave'} at 127.27.196.67:56925)
I20240502 21:35:05.582388  9532 tablet_copy_service.cc:161] P 19f162e164854f2da623a4bb8f9e6bf6: Beginning new tablet copy session on tablet 3d1a5050d3f944969a560994a23700ad from peer f2fdd596a8b44bbc848fe40f9f07fbe0 at {username='slave'} at 127.27.196.67:56925: session id = f2fdd596a8b44bbc848fe40f9f07fbe0-3d1a5050d3f944969a560994a23700ad
I20240502 21:35:05.584362  9533 tablet_copy_source_session.cc:205] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:35:05.585767 24611 ts_tablet_manager.cc:1875] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:35:05.586170  9532 tablet_copy_source_session.cc:205] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:35:05.588068 24583 tablet_copy_client.cc:308] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:43203
I20240502 21:35:05.588510 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1a5050d3f944969a560994a23700ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:05.589201  9399 tablet_copy_service.cc:140] P f2fdd596a8b44bbc848fe40f9f07fbe0: Received BeginTabletCopySession request for tablet 8a2e06253c51405fa9b6bc9e87844760 from peer 525a093ea5bf40dc9bbf9dc39895d670 ({username='slave'} at 127.27.196.66:49721)
I20240502 21:35:05.589373  9399 tablet_copy_service.cc:161] P f2fdd596a8b44bbc848fe40f9f07fbe0: Beginning new tablet copy session on tablet 8a2e06253c51405fa9b6bc9e87844760 from peer 525a093ea5bf40dc9bbf9dc39895d670 at {username='slave'} at 127.27.196.66:49721: session id = 525a093ea5bf40dc9bbf9dc39895d670-8a2e06253c51405fa9b6bc9e87844760
I20240502 21:35:05.592931  9399 tablet_copy_source_session.cc:205] T 8a2e06253c51405fa9b6bc9e87844760 P f2fdd596a8b44bbc848fe40f9f07fbe0: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:35:05.594949 24627 tablet_copy_client.cc:783] T dc1970b9263d41abb575b9d8639c59ae P f2fdd596a8b44bbc848fe40f9f07fbe0: tablet copy: Starting download of 5 data blocks...
I20240502 21:35:05.599584 24583 ts_tablet_manager.cc:1875] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:35:05.601588 24574 tablet_copy_client.cc:783] T 3d1a5050d3f944969a560994a23700ad P f2fdd596a8b44bbc848fe40f9f07fbe0: tablet copy: Starting download of 5 data blocks...
I20240502 21:35:05.608784 24622 tablet_copy_client.cc:308] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:42177
I20240502 21:35:05.620909 24574 tablet_copy_client.cc:647] T 3d1a5050d3f944969a560994a23700ad P f2fdd596a8b44bbc848fe40f9f07fbe0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:35:05.623678 24574 tablet_copy_client.cc:523] T 3d1a5050d3f944969a560994a23700ad P f2fdd596a8b44bbc848fe40f9f07fbe0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:35:05.629846 24627 tablet_copy_client.cc:647] T dc1970b9263d41abb575b9d8639c59ae P f2fdd596a8b44bbc848fe40f9f07fbe0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:35:05.634320 24627 tablet_copy_client.cc:523] T dc1970b9263d41abb575b9d8639c59ae P f2fdd596a8b44bbc848fe40f9f07fbe0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:35:05.635831 24592 tablet_bootstrap.cc:492] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:35:05.636445 24592 tablet_bootstrap.cc:492] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58: Bootstrap complete.
I20240502 21:35:05.636886 24592 ts_tablet_manager.cc:1367] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58: Time spent bootstrapping tablet: real 0.121s	user 0.028s	sys 0.001s
I20240502 21:35:05.638689 24592 raft_consensus.cc:348] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:35:05.638949 24592 raft_consensus.cc:729] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Initialized, Role: FOLLOWER
I20240502 21:35:05.639456 24592 consensus_queue.cc:260] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:35:05.640416 24592 ts_tablet_manager.cc:1398] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 21:35:05.641290  9532 tablet_copy_service.cc:342] P 19f162e164854f2da623a4bb8f9e6bf6: Request end of tablet copy session 5543ddab5b484642b139b7c848932a58-be34ec89876d42b69efe1878c0d9bc19 received from {username='slave'} at 127.27.196.65:48061
I20240502 21:35:05.641515  9532 tablet_copy_service.cc:433] P 19f162e164854f2da623a4bb8f9e6bf6: ending tablet copy session 5543ddab5b484642b139b7c848932a58-be34ec89876d42b69efe1878c0d9bc19 on tablet be34ec89876d42b69efe1878c0d9bc19 with peer 5543ddab5b484642b139b7c848932a58
I20240502 21:35:05.656906 24574 tablet_bootstrap.cc:492] T 3d1a5050d3f944969a560994a23700ad P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap starting.
I20240502 21:35:05.665732 24627 tablet_bootstrap.cc:492] T dc1970b9263d41abb575b9d8639c59ae P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap starting.
I20240502 21:35:05.698879  9128 tablet_copy_service.cc:140] P 5543ddab5b484642b139b7c848932a58: Received BeginTabletCopySession request for tablet 9240163cfac446d2a14dbd9fcbbf9ff7 from peer 4cbb5b06a6a44eb6b2145cd538b01af2 ({username='slave'} at 127.27.196.71:34379)
I20240502 21:35:05.699301  9128 tablet_copy_service.cc:161] P 5543ddab5b484642b139b7c848932a58: Beginning new tablet copy session on tablet 9240163cfac446d2a14dbd9fcbbf9ff7 from peer 4cbb5b06a6a44eb6b2145cd538b01af2 at {username='slave'} at 127.27.196.71:34379: session id = 4cbb5b06a6a44eb6b2145cd538b01af2-9240163cfac446d2a14dbd9fcbbf9ff7
I20240502 21:35:05.700580 24589 tablet_bootstrap.cc:492] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:35:05.701021 24589 tablet_bootstrap.cc:492] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58: Bootstrap complete.
I20240502 21:35:05.701354 24589 ts_tablet_manager.cc:1367] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58: Time spent bootstrapping tablet: real 0.155s	user 0.026s	sys 0.006s
I20240502 21:35:05.701774  9290 maintenance_manager.cc:419] P 525a093ea5bf40dc9bbf9dc39895d670: Scheduling UndoDeltaBlockGCOp(94d2cdf8ae6941668b5f680c558656e8): 4925 bytes on disk
I20240502 21:35:05.702201 24589 raft_consensus.cc:348] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:35:05.702431 24589 raft_consensus.cc:729] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Initialized, Role: FOLLOWER
I20240502 21:35:05.702694 24589 consensus_queue.cc:260] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:35:05.702865  9176 maintenance_manager.cc:643] P 525a093ea5bf40dc9bbf9dc39895d670: UndoDeltaBlockGCOp(94d2cdf8ae6941668b5f680c558656e8) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240502 21:35:05.708563  9128 tablet_copy_source_session.cc:205] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58: Tablet Copy: opened 5 blocks and 1 log segments
W20240502 21:35:05.710680  9039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 35 similar messages]
I20240502 21:35:05.714166 24611 ts_tablet_manager.cc:1888] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 21:35:05.717820 24622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9240163cfac446d2a14dbd9fcbbf9ff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:05.721341 24627 tablet_bootstrap.cc:492] T dc1970b9263d41abb575b9d8639c59ae P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:35:05.721860 24627 tablet_bootstrap.cc:492] T dc1970b9263d41abb575b9d8639c59ae P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap complete.
I20240502 21:35:05.722203 24627 ts_tablet_manager.cc:1367] T dc1970b9263d41abb575b9d8639c59ae P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent bootstrapping tablet: real 0.057s	user 0.018s	sys 0.004s
I20240502 21:35:05.723068 24627 raft_consensus.cc:348] T dc1970b9263d41abb575b9d8639c59ae P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:35:05.723380 24627 raft_consensus.cc:729] T dc1970b9263d41abb575b9d8639c59ae P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Initialized, Role: FOLLOWER
I20240502 21:35:05.724340 24574 tablet_bootstrap.cc:492] T 3d1a5050d3f944969a560994a23700ad P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:35:05.724797 24574 tablet_bootstrap.cc:492] T 3d1a5050d3f944969a560994a23700ad P f2fdd596a8b44bbc848fe40f9f07fbe0: Bootstrap complete.
I20240502 21:35:05.725090 24574 ts_tablet_manager.cc:1367] T 3d1a5050d3f944969a560994a23700ad P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent bootstrapping tablet: real 0.068s	user 0.023s	sys 0.003s
I20240502 21:35:05.725915 24574 raft_consensus.cc:348] T 3d1a5050d3f944969a560994a23700ad P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:35:05.726178 24574 raft_consensus.cc:729] T 3d1a5050d3f944969a560994a23700ad P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fdd596a8b44bbc848fe40f9f07fbe0, State: Initialized, Role: FOLLOWER
I20240502 21:35:05.726522 24627 consensus_queue.cc:260] T dc1970b9263d41abb575b9d8639c59ae P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:35:05.726917 24574 consensus_queue.cc:260] T 3d1a5050d3f944969a560994a23700ad P f2fdd596a8b44bbc848fe40f9f07fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:35:05.727723 24627 ts_tablet_manager.cc:1398] T dc1970b9263d41abb575b9d8639c59ae P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20240502 21:35:05.728242 24574 ts_tablet_manager.cc:1398] T 3d1a5050d3f944969a560994a23700ad P f2fdd596a8b44bbc848fe40f9f07fbe0: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.003s
I20240502 21:35:05.729526  9532 tablet_copy_service.cc:342] P 19f162e164854f2da623a4bb8f9e6bf6: Request end of tablet copy session f2fdd596a8b44bbc848fe40f9f07fbe0-3d1a5050d3f944969a560994a23700ad received from {username='slave'} at 127.27.196.67:56925
I20240502 21:35:05.729691  9532 tablet_copy_service.cc:433] P 19f162e164854f2da623a4bb8f9e6bf6: ending tablet copy session f2fdd596a8b44bbc848fe40f9f07fbe0-3d1a5050d3f944969a560994a23700ad on tablet 3d1a5050d3f944969a560994a23700ad with peer f2fdd596a8b44bbc848fe40f9f07fbe0
I20240502 21:35:05.716979 24589 ts_tablet_manager.cc:1398] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58: Time spent starting tablet: real 0.015s	user 0.003s	sys 0.000s
I20240502 21:35:05.742225  9128 tablet_copy_service.cc:342] P 5543ddab5b484642b139b7c848932a58: Request end of tablet copy session f2fdd596a8b44bbc848fe40f9f07fbe0-dc1970b9263d41abb575b9d8639c59ae received from {username='slave'} at 127.27.196.67:40917
I20240502 21:35:05.743290  9128 tablet_copy_service.cc:433] P 5543ddab5b484642b139b7c848932a58: ending tablet copy session f2fdd596a8b44bbc848fe40f9f07fbe0-dc1970b9263d41abb575b9d8639c59ae on tablet dc1970b9263d41abb575b9d8639c59ae with peer f2fdd596a8b44bbc848fe40f9f07fbe0
W20240502 21:35:05.742913  9039 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): Couldn't send request to peer 6ef0beaf2f6941afb6ea9f9f274517fe. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240502 21:35:05.743940  9399 tablet_copy_service.cc:342] P f2fdd596a8b44bbc848fe40f9f07fbe0: Request end of tablet copy session 5543ddab5b484642b139b7c848932a58-4674e376bf7440cda619784491685b44 received from {username='slave'} at 127.27.196.65:38501
I20240502 21:35:05.744083  9399 tablet_copy_service.cc:433] P f2fdd596a8b44bbc848fe40f9f07fbe0: ending tablet copy session 5543ddab5b484642b139b7c848932a58-4674e376bf7440cda619784491685b44 on tablet 4674e376bf7440cda619784491685b44 with peer 5543ddab5b484642b139b7c848932a58
I20240502 21:35:05.744277 24622 tablet_copy_client.cc:783] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Starting download of 5 data blocks...
I20240502 21:35:05.744659 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c56ac926025e47218e9df4088184a278. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:05.753300  9239 raft_consensus.cc:1203] T 94d2cdf8ae6941668b5f680c558656e8 P 525a093ea5bf40dc9bbf9dc39895d670 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10]   Dedup: 2.10->[]
I20240502 21:35:05.756668 24622 tablet_copy_client.cc:647] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:35:05.759061 24622 tablet_copy_client.cc:523] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:35:05.766638 24593 tablet_bootstrap.cc:492] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:35:05.766914 24622 tablet_bootstrap.cc:492] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap starting.
I20240502 21:35:05.783957 24593 tablet_bootstrap.cc:492] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6: Bootstrap complete.
I20240502 21:35:05.784905 24593 ts_tablet_manager.cc:1367] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6: Time spent bootstrapping tablet: real 0.230s	user 0.039s	sys 0.017s
I20240502 21:35:05.786325 24593 raft_consensus.cc:348] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:35:05.786618 24593 raft_consensus.cc:729] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f162e164854f2da623a4bb8f9e6bf6, State: Initialized, Role: FOLLOWER
I20240502 21:35:05.786875 24611 tablet_copy_client.cc:783] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58: tablet copy: Starting download of 5 data blocks...
I20240502 21:35:05.799988 24611 tablet_copy_client.cc:647] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58: tablet copy: Starting download of 1 WAL segments...
I20240502 21:35:05.803005 24611 tablet_copy_client.cc:523] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:35:05.804940 24622 tablet_bootstrap.cc:492] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:35:05.805487 24622 tablet_bootstrap.cc:492] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2: Bootstrap complete.
I20240502 21:35:05.805851 24622 ts_tablet_manager.cc:1367] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2: Time spent bootstrapping tablet: real 0.039s	user 0.027s	sys 0.004s
I20240502 21:35:05.806617 24593 consensus_queue.cc:260] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:35:05.806797 24622 raft_consensus.cc:348] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:35:05.807328 24622 raft_consensus.cc:729] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbb5b06a6a44eb6b2145cd538b01af2, State: Initialized, Role: FOLLOWER
I20240502 21:35:05.807689 24622 consensus_queue.cc:260] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:35:05.809478 24622 ts_tablet_manager.cc:1398] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 21:35:05.809547 24611 tablet_bootstrap.cc:492] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58: Bootstrap starting.
I20240502 21:35:05.810295 24593 ts_tablet_manager.cc:1398] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6: Time spent starting tablet: real 0.025s	user 0.002s	sys 0.002s
I20240502 21:35:05.810460  9128 tablet_copy_service.cc:342] P 5543ddab5b484642b139b7c848932a58: Request end of tablet copy session 4cbb5b06a6a44eb6b2145cd538b01af2-9240163cfac446d2a14dbd9fcbbf9ff7 received from {username='slave'} at 127.27.196.71:34379
I20240502 21:35:05.810585  9128 tablet_copy_service.cc:433] P 5543ddab5b484642b139b7c848932a58: ending tablet copy session 4cbb5b06a6a44eb6b2145cd538b01af2-9240163cfac446d2a14dbd9fcbbf9ff7 on tablet 9240163cfac446d2a14dbd9fcbbf9ff7 with peer 4cbb5b06a6a44eb6b2145cd538b01af2
I20240502 21:35:05.813086  9128 tablet_copy_service.cc:342] P 5543ddab5b484642b139b7c848932a58: Request end of tablet copy session 19f162e164854f2da623a4bb8f9e6bf6-32c306d724954a16be21a34aa4cc0b5f received from {username='slave'} at 127.27.196.68:53551
I20240502 21:35:05.813221  9128 tablet_copy_service.cc:433] P 5543ddab5b484642b139b7c848932a58: ending tablet copy session 19f162e164854f2da623a4bb8f9e6bf6-32c306d724954a16be21a34aa4cc0b5f on tablet 32c306d724954a16be21a34aa4cc0b5f with peer 19f162e164854f2da623a4bb8f9e6bf6
I20240502 21:35:05.824087  9154 maintenance_manager.cc:419] P 5543ddab5b484642b139b7c848932a58: Scheduling UndoDeltaBlockGCOp(4674e376bf7440cda619784491685b44): 4925 bytes on disk
I20240502 21:35:05.824836  9042 maintenance_manager.cc:643] P 5543ddab5b484642b139b7c848932a58: UndoDeltaBlockGCOp(4674e376bf7440cda619784491685b44) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240502 21:35:05.826181  9154 maintenance_manager.cc:419] P 5543ddab5b484642b139b7c848932a58: Scheduling UndoDeltaBlockGCOp(be34ec89876d42b69efe1878c0d9bc19): 4586 bytes on disk
I20240502 21:35:05.826892  9042 maintenance_manager.cc:643] P 5543ddab5b484642b139b7c848932a58: UndoDeltaBlockGCOp(be34ec89876d42b69efe1878c0d9bc19) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240502 21:35:05.860065  9107 raft_consensus.cc:1203] T be34ec89876d42b69efe1878c0d9bc19 P 5543ddab5b484642b139b7c848932a58 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10]   Dedup: 2.10->[]
W20240502 21:35:05.906912  9039 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.69:42049, user_credentials={real_user=slave}} blocked reactor thread for 86764.3us
I20240502 21:35:05.910936  9508 raft_consensus.cc:1203] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240502 21:35:05.933125 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:35:05.936019  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:05.929703 24583 ts_tablet_manager.cc:1888] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 21:35:05.940117 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2e06253c51405fa9b6bc9e87844760. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:05.944319 24583 tablet_copy_client.cc:783] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670: tablet copy: Starting download of 5 data blocks...
I20240502 21:35:05.944752  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:35:05.948035 24611 tablet_bootstrap.cc:492] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:35:05.950568 24611 tablet_bootstrap.cc:492] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58: Bootstrap complete.
I20240502 21:35:05.952167 24611 ts_tablet_manager.cc:1367] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58: Time spent bootstrapping tablet: real 0.143s	user 0.030s	sys 0.000s
I20240502 21:35:05.954075  9377 raft_consensus.cc:1203] T dc1970b9263d41abb575b9d8639c59ae P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 21:35:05.954123  9426 maintenance_manager.cc:419] P f2fdd596a8b44bbc848fe40f9f07fbe0: Scheduling UndoDeltaBlockGCOp(dc1970b9263d41abb575b9d8639c59ae): 4925 bytes on disk
I20240502 21:35:05.954619 24611 raft_consensus.cc:348] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:35:05.954887 24611 raft_consensus.cc:729] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Initialized, Role: FOLLOWER
I20240502 21:35:05.954825  9314 maintenance_manager.cc:643] P f2fdd596a8b44bbc848fe40f9f07fbe0: UndoDeltaBlockGCOp(dc1970b9263d41abb575b9d8639c59ae) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20240502 21:35:05.956146  9426 maintenance_manager.cc:419] P f2fdd596a8b44bbc848fe40f9f07fbe0: Scheduling UndoDeltaBlockGCOp(3d1a5050d3f944969a560994a23700ad): 4838 bytes on disk
I20240502 21:35:05.956835  9314 maintenance_manager.cc:643] P f2fdd596a8b44bbc848fe40f9f07fbe0: UndoDeltaBlockGCOp(3d1a5050d3f944969a560994a23700ad) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240502 21:35:05.956669 24611 consensus_queue.cc:260] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:35:05.958261 24611 ts_tablet_manager.cc:1398] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240502 21:35:05.959012  9532 tablet_copy_service.cc:342] P 19f162e164854f2da623a4bb8f9e6bf6: Request end of tablet copy session 5543ddab5b484642b139b7c848932a58-c56ac926025e47218e9df4088184a278 received from {username='slave'} at 127.27.196.65:48061
I20240502 21:35:05.959192  9532 tablet_copy_service.cc:433] P 19f162e164854f2da623a4bb8f9e6bf6: ending tablet copy session 5543ddab5b484642b139b7c848932a58-c56ac926025e47218e9df4088184a278 on tablet c56ac926025e47218e9df4088184a278 with peer 5543ddab5b484642b139b7c848932a58
I20240502 21:35:05.964529  9562 maintenance_manager.cc:419] P 19f162e164854f2da623a4bb8f9e6bf6: Scheduling UndoDeltaBlockGCOp(32c306d724954a16be21a34aa4cc0b5f): 4840 bytes on disk
I20240502 21:35:05.965406  9448 maintenance_manager.cc:643] P 19f162e164854f2da623a4bb8f9e6bf6: UndoDeltaBlockGCOp(32c306d724954a16be21a34aa4cc0b5f) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240502 21:35:05.967195 24583 tablet_copy_client.cc:647] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670: tablet copy: Starting download of 1 WAL segments...
I20240502 21:35:05.969792 24583 tablet_copy_client.cc:523] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:35:05.974568 24583 tablet_bootstrap.cc:492] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap starting.
I20240502 21:35:05.995028  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:35:06.027367  9377 raft_consensus.cc:1203] T 3d1a5050d3f944969a560994a23700ad P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 21:35:06.035902 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:06.059516 11411 maintenance_manager.cc:419] P 4cbb5b06a6a44eb6b2145cd538b01af2: Scheduling UndoDeltaBlockGCOp(9240163cfac446d2a14dbd9fcbbf9ff7): 4838 bytes on disk
I20240502 21:35:06.060391 11300 maintenance_manager.cc:643] P 4cbb5b06a6a44eb6b2145cd538b01af2: UndoDeltaBlockGCOp(9240163cfac446d2a14dbd9fcbbf9ff7) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":135,"lbm_reads_lt_1ms":4}
I20240502 21:35:06.057631 11365 raft_consensus.cc:1203] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 21:35:06.063556 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:06.079679  9154 maintenance_manager.cc:419] P 5543ddab5b484642b139b7c848932a58: Scheduling UndoDeltaBlockGCOp(c56ac926025e47218e9df4088184a278): 4925 bytes on disk
I20240502 21:35:06.080746  9042 maintenance_manager.cc:643] P 5543ddab5b484642b139b7c848932a58: UndoDeltaBlockGCOp(c56ac926025e47218e9df4088184a278) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":162,"lbm_reads_lt_1ms":4}
I20240502 21:35:06.092623  9107 raft_consensus.cc:1203] T c56ac926025e47218e9df4088184a278 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 21:35:06.100044  9107 raft_consensus.cc:1203] T 4674e376bf7440cda619784491685b44 P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240502 21:35:06.136534  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:06.161410 24583 tablet_bootstrap.cc:492] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:35:06.164733 24583 tablet_bootstrap.cc:492] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap complete.
I20240502 21:35:06.165197 24583 ts_tablet_manager.cc:1367] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670: Time spent bootstrapping tablet: real 0.191s	user 0.069s	sys 0.041s
I20240502 21:35:06.166447 24583 raft_consensus.cc:348] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:35:06.166760 24583 raft_consensus.cc:729] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Initialized, Role: FOLLOWER
I20240502 21:35:06.175848 24583 consensus_queue.cc:260] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:35:06.179390 24583 ts_tablet_manager.cc:1398] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.000s
I20240502 21:35:06.180968  9399 tablet_copy_service.cc:342] P f2fdd596a8b44bbc848fe40f9f07fbe0: Request end of tablet copy session 525a093ea5bf40dc9bbf9dc39895d670-8a2e06253c51405fa9b6bc9e87844760 received from {username='slave'} at 127.27.196.66:49721
I20240502 21:35:06.181196  9399 tablet_copy_service.cc:433] P f2fdd596a8b44bbc848fe40f9f07fbe0: ending tablet copy session 525a093ea5bf40dc9bbf9dc39895d670-8a2e06253c51405fa9b6bc9e87844760 on tablet 8a2e06253c51405fa9b6bc9e87844760 with peer 525a093ea5bf40dc9bbf9dc39895d670
I20240502 21:35:06.209177  9290 maintenance_manager.cc:419] P 525a093ea5bf40dc9bbf9dc39895d670: Scheduling UndoDeltaBlockGCOp(8a2e06253c51405fa9b6bc9e87844760): 4925 bytes on disk
I20240502 21:35:06.209940  9176 maintenance_manager.cc:643] P 525a093ea5bf40dc9bbf9dc39895d670: UndoDeltaBlockGCOp(8a2e06253c51405fa9b6bc9e87844760) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240502 21:35:06.372562  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:06.378405  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:35:06.381311 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:35:06.428628 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:06.452883  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:35:06.464252  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:06.487665 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:06.495420  9239 raft_consensus.cc:1203] T 8a2e06253c51405fa9b6bc9e87844760 P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240502 21:35:06.508586  9239 consensus_queue.cc:237] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:06.510854 11093 raft_consensus.cc:1261] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:35:06.511556 24478 consensus_queue.cc:1035] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:35:06.542546 24743 raft_consensus.cc:2932] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:35:06.543442 11093 raft_consensus.cc:2932] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:35:06.549046  8955 catalog_manager.cc:5484] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 reported cstate change: config changed from index 7 to 8, VOTER f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67) evicted. New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:35:06.579731  9356 tablet_service.cc:1514] Processing DeleteTablet for tablet 3caa36212ba444aa88d153ff9f90122e with delete_type TABLET_DATA_TOMBSTONED (TS f2fdd596a8b44bbc848fe40f9f07fbe0 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:52648
I20240502 21:35:06.581115 24750 tablet_replica.cc:330] T 3caa36212ba444aa88d153ff9f90122e P f2fdd596a8b44bbc848fe40f9f07fbe0: stopping tablet replica
I20240502 21:35:06.581630 24750 raft_consensus.cc:2229] T 3caa36212ba444aa88d153ff9f90122e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:35:06.581940 24750 raft_consensus.cc:2258] T 3caa36212ba444aa88d153ff9f90122e P f2fdd596a8b44bbc848fe40f9f07fbe0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:35:06.583211 24750 ts_tablet_manager.cc:1875] T 3caa36212ba444aa88d153ff9f90122e P f2fdd596a8b44bbc848fe40f9f07fbe0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:35:06.587945  9239 consensus_queue.cc:237] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:35:06.590514 11093 raft_consensus.cc:1261] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Refusing update from remote peer 525a093ea5bf40dc9bbf9dc39895d670: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:35:06.591154 24746 consensus_queue.cc:1035] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:35:06.593611 24744 raft_consensus.cc:2932] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } }
W20240502 21:35:06.598733  9172 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Couldn't send request to peer 5543ddab5b484642b139b7c848932a58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3caa36212ba444aa88d153ff9f90122e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:06.598536 11093 raft_consensus.cc:2932] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } }
I20240502 21:35:06.603700  8949 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 3caa36212ba444aa88d153ff9f90122e with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 21:35:06.606370  8955 catalog_manager.cc:5484] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 reported cstate change: config changed from index 8 to 9, VOTER 5543ddab5b484642b139b7c848932a58 (127.27.196.65) added. New cstate: current_term: 1 leader_uuid: "525a093ea5bf40dc9bbf9dc39895d670" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } }
I20240502 21:35:06.615721 24750 ts_tablet_manager.cc:1888] T 3caa36212ba444aa88d153ff9f90122e P f2fdd596a8b44bbc848fe40f9f07fbe0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 21:35:06.615896 24750 log.cc:1196] T 3caa36212ba444aa88d153ff9f90122e P f2fdd596a8b44bbc848fe40f9f07fbe0: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/wals/3caa36212ba444aa88d153ff9f90122e
I20240502 21:35:06.616820  8948 catalog_manager.cc:4830] TS f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): tablet 3caa36212ba444aa88d153ff9f90122e (table rebalance_test_table_9 [id=16e8100140034581b2a773dcf740998d]) successfully deleted
I20240502 21:35:06.666193  9107 consensus_queue.cc:237] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:35:06.669092 11365 raft_consensus.cc:1261] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:35:06.669816 24526 consensus_queue.cc:1035] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:35:06.673027 24672 raft_consensus.cc:2932] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:35:06.673663 11365 raft_consensus.cc:2932] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:35:06.679278  8956 catalog_manager.cc:5484] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2 reported cstate change: config changed from index 7 to 8, VOTER 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68) evicted. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:35:06.683722  9490 tablet_service.cc:1514] Processing DeleteTablet for tablet 9240163cfac446d2a14dbd9fcbbf9ff7 with delete_type TABLET_DATA_TOMBSTONED (TS 19f162e164854f2da623a4bb8f9e6bf6 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:44784
I20240502 21:35:06.684440 24772 tablet_replica.cc:330] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 19f162e164854f2da623a4bb8f9e6bf6: stopping tablet replica
I20240502 21:35:06.684731 24772 raft_consensus.cc:2229] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:35:06.685041 24772 raft_consensus.cc:2258] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:35:06.685320  9107 consensus_queue.cc:237] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:06.686012 24772 ts_tablet_manager.cc:1875] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 19f162e164854f2da623a4bb8f9e6bf6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:35:06.687680 11365 raft_consensus.cc:1261] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 21:35:06.687752  9039 consensus_peers.cc:485] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 -> Peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Couldn't send request to peer d10e9e156ba74a97a221c85b68ab8bc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9240163cfac446d2a14dbd9fcbbf9ff7. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:06.688524 24672 consensus_queue.cc:1035] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:35:06.692836 24672 raft_consensus.cc:2932] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:35:06.694293 11365 raft_consensus.cc:2932] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:35:06.698557  8945 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 9240163cfac446d2a14dbd9fcbbf9ff7 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 21:35:06.702981  8956 catalog_manager.cc:5484] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index 8 to 9, VOTER d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69) added. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:35:06.708526 24772 ts_tablet_manager.cc:1888] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 19f162e164854f2da623a4bb8f9e6bf6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 21:35:06.708726 24772 log.cc:1196] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 19f162e164854f2da623a4bb8f9e6bf6: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/wals/9240163cfac446d2a14dbd9fcbbf9ff7
I20240502 21:35:06.709611  8945 catalog_manager.cc:4830] TS 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): tablet 9240163cfac446d2a14dbd9fcbbf9ff7 (table rebalance_test_table_0 [id=445d2e6ec7154b80abec5fff346b8f5e]) successfully deleted
I20240502 21:35:06.745409 24672 raft_consensus.cc:998] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58: Attempting to remove follower 6ef0beaf2f6941afb6ea9f9f274517fe from the Raft config. Reason: Leader has been unable to successfully communicate with peer 6ef0beaf2f6941afb6ea9f9f274517fe for more than 300 seconds (300.127s)
I20240502 21:35:06.746215 24672 consensus_queue.cc:237] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:35:06.748895 11365 raft_consensus.cc:1261] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:35:06.749533 24778 consensus_queue.cc:1035] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:35:06.752538 24672 raft_consensus.cc:2932] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:35:06.753237 11365 raft_consensus.cc:2932] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:35:06.760030  8955 catalog_manager.cc:5484] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 reported cstate change: config changed from index 7 to 8, VOTER 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70) evicted. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } }
I20240502 21:35:06.767026  9107 consensus_queue.cc:237] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
W20240502 21:35:06.769676  9041 consensus_peers.cc:485] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 -> Peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Couldn't send request to peer 525a093ea5bf40dc9bbf9dc39895d670. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240502 21:35:06.770478  8946 catalog_manager.cc:4573] TS 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet cdf72d25f0f847e5998acbc53979e64d: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:35:06.770768 11365 raft_consensus.cc:1261] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Refusing update from remote peer 5543ddab5b484642b139b7c848932a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:35:06.771543 24778 consensus_queue.cc:1035] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:35:06.774709 24672 raft_consensus.cc:2932] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } }
I20240502 21:35:06.775370 11365 raft_consensus.cc:2932] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } }
I20240502 21:35:06.781338  8945 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet cdf72d25f0f847e5998acbc53979e64d with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 21:35:06.781780  8955 catalog_manager.cc:5484] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 reported cstate change: config changed from index 8 to 9, VOTER 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66) added. New cstate: current_term: 1 leader_uuid: "5543ddab5b484642b139b7c848932a58" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } }
I20240502 21:35:07.001030 24830 ts_tablet_manager.cc:902] T 3caa36212ba444aa88d153ff9f90122e P 5543ddab5b484642b139b7c848932a58: Initiating tablet copy from peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:35:07.035830  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:35:07.065543  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:07.086083 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:07.087893  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:35:07.105316  9444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.65:42177, user_credentials={real_user=slave}} blocked reactor thread for 60183.5us
I20240502 21:35:07.145993 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:07.162400  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:07.211737 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:35:07.211714 24830 tablet_copy_client.cc:308] T 3caa36212ba444aa88d153ff9f90122e P 5543ddab5b484642b139b7c848932a58: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:39071
I20240502 21:35:07.215502 24852 ts_tablet_manager.cc:902] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4: Initiating tablet copy from peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:35:07.233187  9263 tablet_copy_service.cc:140] P 525a093ea5bf40dc9bbf9dc39895d670: Received BeginTabletCopySession request for tablet 3caa36212ba444aa88d153ff9f90122e from peer 5543ddab5b484642b139b7c848932a58 ({username='slave'} at 127.27.196.65:48383)
I20240502 21:35:07.233402  9263 tablet_copy_service.cc:161] P 525a093ea5bf40dc9bbf9dc39895d670: Beginning new tablet copy session on tablet 3caa36212ba444aa88d153ff9f90122e from peer 5543ddab5b484642b139b7c848932a58 at {username='slave'} at 127.27.196.65:48383: session id = 5543ddab5b484642b139b7c848932a58-3caa36212ba444aa88d153ff9f90122e
I20240502 21:35:07.237655  9263 tablet_copy_source_session.cc:205] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:35:07.242280 24830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3caa36212ba444aa88d153ff9f90122e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:07.267449 24830 tablet_copy_client.cc:783] T 3caa36212ba444aa88d153ff9f90122e P 5543ddab5b484642b139b7c848932a58: tablet copy: Starting download of 5 data blocks...
I20240502 21:35:07.277161 24830 tablet_copy_client.cc:647] T 3caa36212ba444aa88d153ff9f90122e P 5543ddab5b484642b139b7c848932a58: tablet copy: Starting download of 1 WAL segments...
I20240502 21:35:07.279475 24830 tablet_copy_client.cc:523] T 3caa36212ba444aa88d153ff9f90122e P 5543ddab5b484642b139b7c848932a58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:35:07.283437 24852 tablet_copy_client.cc:308] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:42177
I20240502 21:35:07.290719 24830 tablet_bootstrap.cc:492] T 3caa36212ba444aa88d153ff9f90122e P 5543ddab5b484642b139b7c848932a58: Bootstrap starting.
I20240502 21:35:07.306013  9128 tablet_copy_service.cc:140] P 5543ddab5b484642b139b7c848932a58: Received BeginTabletCopySession request for tablet 9240163cfac446d2a14dbd9fcbbf9ff7 from peer d10e9e156ba74a97a221c85b68ab8bc4 ({username='slave'} at 127.27.196.69:35099)
I20240502 21:35:07.306227  9128 tablet_copy_service.cc:161] P 5543ddab5b484642b139b7c848932a58: Beginning new tablet copy session on tablet 9240163cfac446d2a14dbd9fcbbf9ff7 from peer d10e9e156ba74a97a221c85b68ab8bc4 at {username='slave'} at 127.27.196.69:35099: session id = d10e9e156ba74a97a221c85b68ab8bc4-9240163cfac446d2a14dbd9fcbbf9ff7
I20240502 21:35:07.309897  9128 tablet_copy_source_session.cc:205] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 5543ddab5b484642b139b7c848932a58: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:35:07.312234 24852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9240163cfac446d2a14dbd9fcbbf9ff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:07.322604 24852 tablet_copy_client.cc:783] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Starting download of 5 data blocks...
I20240502 21:35:07.330415 24852 tablet_copy_client.cc:647] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:35:07.332674 24852 tablet_copy_client.cc:523] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:35:07.337734 24852 tablet_bootstrap.cc:492] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap starting.
I20240502 21:35:07.374328 24865 ts_tablet_manager.cc:902] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670: Initiating tablet copy from peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:35:07.401640 24852 tablet_bootstrap.cc:492] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:35:07.402297 24852 tablet_bootstrap.cc:492] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4: Bootstrap complete.
I20240502 21:35:07.402787 24852 ts_tablet_manager.cc:1367] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4: Time spent bootstrapping tablet: real 0.065s	user 0.050s	sys 0.013s
I20240502 21:35:07.403836 24852 raft_consensus.cc:348] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:07.404232 24852 raft_consensus.cc:729] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d10e9e156ba74a97a221c85b68ab8bc4, State: Initialized, Role: FOLLOWER
I20240502 21:35:07.404600 24852 consensus_queue.cc:260] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:07.405364 24852 ts_tablet_manager.cc:1398] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:35:07.406380  9128 tablet_copy_service.cc:342] P 5543ddab5b484642b139b7c848932a58: Request end of tablet copy session d10e9e156ba74a97a221c85b68ab8bc4-9240163cfac446d2a14dbd9fcbbf9ff7 received from {username='slave'} at 127.27.196.69:35099
I20240502 21:35:07.406518  9128 tablet_copy_service.cc:433] P 5543ddab5b484642b139b7c848932a58: ending tablet copy session d10e9e156ba74a97a221c85b68ab8bc4-9240163cfac446d2a14dbd9fcbbf9ff7 on tablet 9240163cfac446d2a14dbd9fcbbf9ff7 with peer d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:35:07.439491 24865 tablet_copy_client.cc:308] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:42177
I20240502 21:35:07.440680  9128 tablet_copy_service.cc:140] P 5543ddab5b484642b139b7c848932a58: Received BeginTabletCopySession request for tablet cdf72d25f0f847e5998acbc53979e64d from peer 525a093ea5bf40dc9bbf9dc39895d670 ({username='slave'} at 127.27.196.66:48979)
I20240502 21:35:07.440920  9128 tablet_copy_service.cc:161] P 5543ddab5b484642b139b7c848932a58: Beginning new tablet copy session on tablet cdf72d25f0f847e5998acbc53979e64d from peer 525a093ea5bf40dc9bbf9dc39895d670 at {username='slave'} at 127.27.196.66:48979: session id = 525a093ea5bf40dc9bbf9dc39895d670-cdf72d25f0f847e5998acbc53979e64d
I20240502 21:35:07.445372  9128 tablet_copy_source_session.cc:205] T cdf72d25f0f847e5998acbc53979e64d P 5543ddab5b484642b139b7c848932a58: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:35:07.447458 24865 ts_tablet_manager.cc:1875] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:35:07.457754 24865 ts_tablet_manager.cc:1888] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 21:35:07.458107 24865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdf72d25f0f847e5998acbc53979e64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:07.462478 24865 tablet_copy_client.cc:783] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670: tablet copy: Starting download of 5 data blocks...
I20240502 21:35:07.482121 24865 tablet_copy_client.cc:647] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670: tablet copy: Starting download of 1 WAL segments...
I20240502 21:35:07.484417 24865 tablet_copy_client.cc:523] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:35:07.490882 24865 tablet_bootstrap.cc:492] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap starting.
I20240502 21:35:07.498283 24830 tablet_bootstrap.cc:492] T 3caa36212ba444aa88d153ff9f90122e P 5543ddab5b484642b139b7c848932a58: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:35:07.498836 24830 tablet_bootstrap.cc:492] T 3caa36212ba444aa88d153ff9f90122e P 5543ddab5b484642b139b7c848932a58: Bootstrap complete.
I20240502 21:35:07.499253 24830 ts_tablet_manager.cc:1367] T 3caa36212ba444aa88d153ff9f90122e P 5543ddab5b484642b139b7c848932a58: Time spent bootstrapping tablet: real 0.209s	user 0.068s	sys 0.027s
I20240502 21:35:07.500459 24830 raft_consensus.cc:348] T 3caa36212ba444aa88d153ff9f90122e P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:35:07.501165 24830 raft_consensus.cc:729] T 3caa36212ba444aa88d153ff9f90122e P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5543ddab5b484642b139b7c848932a58, State: Initialized, Role: FOLLOWER
I20240502 21:35:07.501596 24830 consensus_queue.cc:260] T 3caa36212ba444aa88d153ff9f90122e P 5543ddab5b484642b139b7c848932a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:35:07.503382 24830 ts_tablet_manager.cc:1398] T 3caa36212ba444aa88d153ff9f90122e P 5543ddab5b484642b139b7c848932a58: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 21:35:07.504170  9263 tablet_copy_service.cc:342] P 525a093ea5bf40dc9bbf9dc39895d670: Request end of tablet copy session 5543ddab5b484642b139b7c848932a58-3caa36212ba444aa88d153ff9f90122e received from {username='slave'} at 127.27.196.65:48383
I20240502 21:35:07.504359  9263 tablet_copy_service.cc:433] P 525a093ea5bf40dc9bbf9dc39895d670: ending tablet copy session 5543ddab5b484642b139b7c848932a58-3caa36212ba444aa88d153ff9f90122e on tablet 3caa36212ba444aa88d153ff9f90122e with peer 5543ddab5b484642b139b7c848932a58
I20240502 21:35:07.509119 11140 maintenance_manager.cc:419] P d10e9e156ba74a97a221c85b68ab8bc4: Scheduling UndoDeltaBlockGCOp(9240163cfac446d2a14dbd9fcbbf9ff7): 4838 bytes on disk
I20240502 21:35:07.510349 11029 maintenance_manager.cc:643] P d10e9e156ba74a97a221c85b68ab8bc4: UndoDeltaBlockGCOp(9240163cfac446d2a14dbd9fcbbf9ff7) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":131,"lbm_reads_lt_1ms":4}
I20240502 21:35:07.523612 24865 tablet_bootstrap.cc:492] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:35:07.524344 24865 tablet_bootstrap.cc:492] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670: Bootstrap complete.
I20240502 21:35:07.524718 24865 ts_tablet_manager.cc:1367] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670: Time spent bootstrapping tablet: real 0.034s	user 0.029s	sys 0.002s
I20240502 21:35:07.525734 24865 raft_consensus.cc:348] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:35:07.526014 24865 raft_consensus.cc:729] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525a093ea5bf40dc9bbf9dc39895d670, State: Initialized, Role: FOLLOWER
I20240502 21:35:07.526309 24865 consensus_queue.cc:260] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:35:07.527968 24865 ts_tablet_manager.cc:1398] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 21:35:07.528921  9128 tablet_copy_service.cc:342] P 5543ddab5b484642b139b7c848932a58: Request end of tablet copy session 525a093ea5bf40dc9bbf9dc39895d670-cdf72d25f0f847e5998acbc53979e64d received from {username='slave'} at 127.27.196.66:48979
I20240502 21:35:07.529067  9128 tablet_copy_service.cc:433] P 5543ddab5b484642b139b7c848932a58: ending tablet copy session 525a093ea5bf40dc9bbf9dc39895d670-cdf72d25f0f847e5998acbc53979e64d on tablet cdf72d25f0f847e5998acbc53979e64d with peer 525a093ea5bf40dc9bbf9dc39895d670
I20240502 21:35:07.581240  9107 raft_consensus.cc:1203] T 3caa36212ba444aa88d153ff9f90122e P 5543ddab5b484642b139b7c848932a58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240502 21:35:07.591934  9154 maintenance_manager.cc:419] P 5543ddab5b484642b139b7c848932a58: Scheduling UndoDeltaBlockGCOp(3caa36212ba444aa88d153ff9f90122e): 4923 bytes on disk
I20240502 21:35:07.592720  9042 maintenance_manager.cc:643] P 5543ddab5b484642b139b7c848932a58: UndoDeltaBlockGCOp(3caa36212ba444aa88d153ff9f90122e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
I20240502 21:35:07.634344  9490 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:07.654356 11343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:35:07.666061  9085 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:35:07.692615  9221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:35:07.705773 11073 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:07.718319  9290 maintenance_manager.cc:419] P 525a093ea5bf40dc9bbf9dc39895d670: Scheduling UndoDeltaBlockGCOp(cdf72d25f0f847e5998acbc53979e64d): 4840 bytes on disk
I20240502 21:35:07.719005  9176 maintenance_manager.cc:643] P 525a093ea5bf40dc9bbf9dc39895d670: UndoDeltaBlockGCOp(cdf72d25f0f847e5998acbc53979e64d) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240502 21:35:07.730281  9239 raft_consensus.cc:1203] T cdf72d25f0f847e5998acbc53979e64d P 525a093ea5bf40dc9bbf9dc39895d670 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240502 21:35:07.780988 11476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:07.807013  9356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:07.822587 11093 raft_consensus.cc:1203] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
W20240502 21:35:07.850934  9313 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.66:39071, user_credentials={real_user=slave}} blocked reactor thread for 53772.2us
W20240502 21:35:07.990132  8946 catalog_manager.cc:4573] TS 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet cdf72d25f0f847e5998acbc53979e64d: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
I20240502 21:35:08.216259 11551 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240502 21:29:08.464905 11557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:08.465871 11557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:08.543444 11557 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:29:12.867969 11557 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 21:29:15.105574 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f52c6c3804274777abfa00bb33ae9b16: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:15.105803 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e8eef05f850243d0bdc7e1899f8c6139: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:15.106000 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 445d2e6ec7154b80abec5fff346b8f5e: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:15.106161 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a237d7bda970485480e98020c6ac738f: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:15.106277 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef412fa8fe9249729cb99879a5c8343a: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:15.106386 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f52c6c3804274777abfa00bb33ae9b16: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:15.106509 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c0a0ef39a2a246449eb582cc1d993484: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:15.106654 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 265d5656d7c048aa9795a906661b8ea8: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:15.106748 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16e8100140034581b2a773dcf740998d: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:15.106904 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1b17408dbab9492bba0ed50efd6ef43f: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:15.107014 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5802da0404e74a07bca290cdc59b279b: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:15.107163 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f52c6c3804274777abfa00bb33ae9b16: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:15.107250 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c0a0ef39a2a246449eb582cc1d993484: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:15.107349 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e8eef05f850243d0bdc7e1899f8c6139: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:15.107437 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 265d5656d7c048aa9795a906661b8ea8: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server 4cbb5b06a6a44eb6b2145cd538b01af2
I20240502 21:29:15.559861 11557 rebalancer_tool.cc:1074] tablet 07d693b680a54e12871121737e3da744: '525a093ea5bf40dc9bbf9dc39895d670' -> 'd10e9e156ba74a97a221c85b68ab8bc4' move scheduled
I20240502 21:29:16.372223 11557 rebalancer_tool.cc:1074] tablet aeee59c5512d448d8f986f959dce1cc8: '19f162e164854f2da623a4bb8f9e6bf6' -> '4cbb5b06a6a44eb6b2145cd538b01af2' move scheduled
I20240502 21:29:16.949425 11557 rebalancer_tool.cc:1084] tablet 41ee481a3ece47a0905e93c66dbf949d: '5543ddab5b484642b139b7c848932a58' -> '6ef0beaf2f6941afb6ea9f9f274517fe' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240502 21:29:16.949636 11557 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 21:29:19.054795 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e8eef05f850243d0bdc7e1899f8c6139: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:19.055159 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16e8100140034581b2a773dcf740998d: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:19.055310 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 445d2e6ec7154b80abec5fff346b8f5e: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:19.055573 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e8eef05f850243d0bdc7e1899f8c6139: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:19.055719 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 265d5656d7c048aa9795a906661b8ea8: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:19.055835 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16e8100140034581b2a773dcf740998d: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:19.055982 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1b17408dbab9492bba0ed50efd6ef43f: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:19.056165 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef412fa8fe9249729cb99879a5c8343a: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:19.056290 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef412fa8fe9249729cb99879a5c8343a: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:19.056449 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f52c6c3804274777abfa00bb33ae9b16: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:19.056614 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 445d2e6ec7154b80abec5fff346b8f5e: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:19.056741 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c0a0ef39a2a246449eb582cc1d993484: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:19.056892 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e8eef05f850243d0bdc7e1899f8c6139: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:19.057020 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 265d5656d7c048aa9795a906661b8ea8: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:19.057161 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5802da0404e74a07bca290cdc59b279b: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:19.057269 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16e8100140034581b2a773dcf740998d: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:19.057410 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1b17408dbab9492bba0ed50efd6ef43f: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:29:19.287253 11557 rebalancer_tool.cc:1074] tablet 58c7004ceb044f5fa9dec54910046b69: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '' move scheduled
I20240502 21:29:19.921077 11557 rebalancer_tool.cc:1074] tablet f0259f01acb249dd99c662151dc21916: '5543ddab5b484642b139b7c848932a58' -> '4cbb5b06a6a44eb6b2145cd538b01af2' move scheduled
I20240502 21:29:20.541604 11557 rebalancer_tool.cc:1074] tablet f82c35077224414cb284472d49d7de4f: '19f162e164854f2da623a4bb8f9e6bf6' -> 'e0de0545fa0547d3929111e990e6d1b2' move scheduled
I20240502 21:29:21.419987 11557 rebalancer_tool.cc:1074] tablet dc1970b9263d41abb575b9d8639c59ae: '525a093ea5bf40dc9bbf9dc39895d670' -> '6ef0beaf2f6941afb6ea9f9f274517fe' move scheduled
I20240502 21:29:21.793258 11557 rebalancer_tool.cc:1084] tablet f9ae8fac693d4d68b6741f2509e1265e: '5543ddab5b484642b139b7c848932a58' -> 'd10e9e156ba74a97a221c85b68ab8bc4' move ignored: Illegal state: tablet f9ae8fac693d4d68b6741f2509e1265e: replica d10e9e156ba74a97a221c85b68ab8bc4 is already present
I20240502 21:29:21.793488 11557 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 21:29:23.692706 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1b17408dbab9492bba0ed50efd6ef43f: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:23.693010 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef412fa8fe9249729cb99879a5c8343a: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:23.693185 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5802da0404e74a07bca290cdc59b279b: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:23.693331 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f52c6c3804274777abfa00bb33ae9b16: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:23.693436 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a237d7bda970485480e98020c6ac738f: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:23.693626 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 445d2e6ec7154b80abec5fff346b8f5e: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:23.693833 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c0a0ef39a2a246449eb582cc1d993484: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:23.693953 11557 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef412fa8fe9249729cb99879a5c8343a: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server 4cbb5b06a6a44eb6b2145cd538b01af2
I20240502 21:29:24.053833 11557 rebalancer_tool.cc:1074] tablet 4d1a31513bb64578a85adf1f622e5bc0: '6ef0beaf2f6941afb6ea9f9f274517fe' -> '' move scheduled
I20240502 21:29:24.341977 11557 rebalancer_tool.cc:1074] tablet 51936fb1f678458a8795f8246590addb: '5543ddab5b484642b139b7c848932a58' -> '' move scheduled
I20240502 21:29:24.803860 11557 rebalancer_tool.cc:1074] tablet 8b4486f03fd9474aa0bb2cb4c9149437: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '' move scheduled
I20240502 21:29:25.138367 11557 rebalancer_tool.cc:1074] tablet 87f35aff2b104c01a9d9d48a7e9c9b23: '19f162e164854f2da623a4bb8f9e6bf6' -> '' move scheduled
W20240502 21:29:31.472620 11656 kernel_stack_watchdog.cc:198] Thread 11557 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 340ms:
Kernel stack:
(could not read kernel stack)
User stack:
    @     0x7f57a58c1980  (unknown) at ??:0
    @           0x90ef77  __sanitizer::internal_sched_yield() at ??:?
    @           0x87462d  __interceptor_pthread_create at ??:?
    @     0x7f57a252fd11  kudu::Thread::StartThread() at ??:0
    @           0xa4d187  kudu::Thread::Create() at ??:?
    @     0x7f57a25507ab  kudu::ThreadPool::CreateThread() at ??:0
    @     0x7f57a254c3ad  kudu::ThreadPool::DoSubmit() at ??:0
    @     0x7f57a25514ff  kudu::ThreadPool::Submit() at ??:0
    @     0x7f57b6a81eaa  kudu::tools::RemoteKsckCluster::RetrieveTablesList() at ??:0
    @     0x7f57b6a05647  kudu::tools::KsckCluster::FetchTableAndTabletInfo() at ??:0
    @     0x7f57b69f0d91  kudu::tools::Ksck::FetchTableAndTabletInfo() at ??:0
    @     0x7f57b7352aac  kudu::tools::DoKsckForTablet() at ??:0
    @     0x7f57b73515c6  kudu::tools::CheckCompleteMove() at ??:0
    @     0x7f57b72d61df  kudu::tools::RebalancerTool::AlgoBasedRunner::UpdateMovesInProgressStatus() at ??:0
    @     0x7f57b72cd82b  kudu::tools::RebalancerTool::RunWith() at ??:0
    @     0x7f57b72ca14c  kudu::tools::RebalancerTool::Run() at ??:0
W20240502 21:29:31.496158 11557 thread.cc:636] RemoteKsckCluster-fetch [worker] (thread pool) Time spent creating pthread: real 0.593s	user 0.131s	sys 0.209s
W20240502 21:29:31.496515 11557 thread.cc:602] RemoteKsckCluster-fetch [worker] (thread pool) Time spent starting thread: real 0.594s	user 0.131s	sys 0.209s
I20240502 21:29:44.751693 11557 rebalancer_tool.cc:1136] tablet f0259f01acb249dd99c662151dc21916: '5543ddab5b484642b139b7c848932a58' -> '4cbb5b06a6a44eb6b2145cd538b01af2' move completed: OK
W20240502 21:29:46.681474 16675 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.27.196.68:40931, user_credentials={real_user=slave}} blocked reactor thread for 50101.3us
I20240502 21:29:55.420763 11557 rebalancer_tool.cc:1136] tablet 51936fb1f678458a8795f8246590addb: '5543ddab5b484642b139b7c848932a58' -> '' move completed: OK
I20240502 21:30:00.045960 11557 rebalancer_tool.cc:1136] tablet aeee59c5512d448d8f986f959dce1cc8: '19f162e164854f2da623a4bb8f9e6bf6' -> '4cbb5b06a6a44eb6b2145cd538b01af2' move completed: OK
I20240502 21:30:00.160957 11557 rebalancer_tool.cc:1136] tablet 8b4486f03fd9474aa0bb2cb4c9149437: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '' move completed: OK
I20240502 21:30:00.246107 11557 rebalancer_tool.cc:1136] tablet 07d693b680a54e12871121737e3da744: '525a093ea5bf40dc9bbf9dc39895d670' -> 'd10e9e156ba74a97a221c85b68ab8bc4' move completed: OK
I20240502 21:30:00.317159 11557 rebalancer_tool.cc:1136] tablet 58c7004ceb044f5fa9dec54910046b69: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '' move completed: OK
I20240502 21:30:00.686720 11557 rebalancer_tool.cc:1136] tablet 87f35aff2b104c01a9d9d48a7e9c9b23: '19f162e164854f2da623a4bb8f9e6bf6' -> '' move completed: OK
I20240502 21:30:01.035511 11557 rebalancer_tool.cc:1136] tablet f82c35077224414cb284472d49d7de4f: '19f162e164854f2da623a4bb8f9e6bf6' -> 'e0de0545fa0547d3929111e990e6d1b2' move completed: OK
I20240502 21:30:01.088780 11557 rebalancer_tool.cc:1136] tablet dc1970b9263d41abb575b9d8639c59ae: '525a093ea5bf40dc9bbf9dc39895d670' -> '6ef0beaf2f6941afb6ea9f9f274517fe' move completed: OK
I20240502 21:30:01.197013 11557 rebalancer_tool.cc:1074] tablet be34ec89876d42b69efe1878c0d9bc19: '5543ddab5b484642b139b7c848932a58' -> '' move scheduled
I20240502 21:30:01.440935 11557 rebalancer_tool.cc:1074] tablet f9ae8fac693d4d68b6741f2509e1265e: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '' move scheduled
I20240502 21:30:01.944697 11557 rebalancer_tool.cc:1074] tablet 8129d13163f64368b731ba4eaeaaa81c: '19f162e164854f2da623a4bb8f9e6bf6' -> '' move scheduled
I20240502 21:30:02.216677 11557 rebalancer_tool.cc:1074] tablet 3d1a5050d3f944969a560994a23700ad: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move scheduled
I20240502 21:30:02.292428 11557 rebalancer_tool.cc:1136] tablet 3d1a5050d3f944969a560994a23700ad: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move completed: OK
I20240502 21:30:02.384486 11557 rebalancer_tool.cc:1136] tablet 8129d13163f64368b731ba4eaeaaa81c: '19f162e164854f2da623a4bb8f9e6bf6' -> '' move completed: OK
I20240502 21:30:02.520331 11557 rebalancer_tool.cc:1136] tablet f9ae8fac693d4d68b6741f2509e1265e: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '' move completed: OK
I20240502 21:30:02.634667 11557 rebalancer_tool.cc:1136] tablet be34ec89876d42b69efe1878c0d9bc19: '5543ddab5b484642b139b7c848932a58' -> '' move completed: OK
I20240502 21:30:05.142697 11557 rebalancer_tool.cc:1074] tablet 8a2e06253c51405fa9b6bc9e87844760: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move scheduled
I20240502 21:30:05.305349 11557 rebalancer_tool.cc:1074] tablet 32c306d724954a16be21a34aa4cc0b5f: '19f162e164854f2da623a4bb8f9e6bf6' -> '' move scheduled
I20240502 21:30:05.512926 11557 rebalancer_tool.cc:1074] tablet 3caa36212ba444aa88d153ff9f90122e: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '' move scheduled
I20240502 21:30:05.739591 11557 rebalancer_tool.cc:1074] tablet a2f3edaf1c874cefb820f027f90d1185: '5543ddab5b484642b139b7c848932a58' -> '' move scheduled
I20240502 21:30:05.963578 11557 rebalancer_tool.cc:1136] tablet a2f3edaf1c874cefb820f027f90d1185: '5543ddab5b484642b139b7c848932a58' -> '' move completed: OK
I20240502 21:30:07.896220 11557 rebalancer_tool.cc:1136] tablet 32c306d724954a16be21a34aa4cc0b5f: '19f162e164854f2da623a4bb8f9e6bf6' -> '' move completed: OK
I20240502 21:30:08.187492 11557 rebalancer_tool.cc:1136] tablet 8a2e06253c51405fa9b6bc9e87844760: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move completed: OK
I20240502 21:30:08.355624 11557 rebalancer_tool.cc:1136] tablet 4d1a31513bb64578a85adf1f622e5bc0: '6ef0beaf2f6941afb6ea9f9f274517fe' -> '' move completed: OK
I20240502 21:30:08.423028 11557 rebalancer_tool.cc:1074] tablet de3287296c4547688388388dd4c7e638: '5543ddab5b484642b139b7c848932a58' -> '' move scheduled
I20240502 21:30:08.614359 11557 rebalancer_tool.cc:1074] tablet cdf72d25f0f847e5998acbc53979e64d: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move scheduled
I20240502 21:30:08.813311 11557 rebalancer_tool.cc:1074] tablet cffcb4f7f7754262beeb06168097f6dc: '5543ddab5b484642b139b7c848932a58' -> '' move scheduled
I20240502 21:30:08.914999 11557 rebalancer_tool.cc:1074] tablet a7c6b881c9ef4cdca787f065778e526f: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move scheduled
I20240502 21:30:09.254827 11557 rebalancer_tool.cc:1136] tablet a7c6b881c9ef4cdca787f065778e526f: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move completed: OK
I20240502 21:30:09.461256 11557 rebalancer_tool.cc:1136] tablet cffcb4f7f7754262beeb06168097f6dc: '5543ddab5b484642b139b7c848932a58' -> '' move completed: OK
I20240502 21:30:09.669507 11557 rebalancer_tool.cc:1136] tablet cdf72d25f0f847e5998acbc53979e64d: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move completed: OK
I20240502 21:30:14.025707 11557 rebalancer_tool.cc:1136] tablet de3287296c4547688388388dd4c7e638: '5543ddab5b484642b139b7c848932a58' -> '' move completed: OK
I20240502 21:35:06.789144 11557 rebalancer_tool.cc:1136] tablet 3caa36212ba444aa88d153ff9f90122e: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '' move completed: OK
Illegal state: tablet server 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): unacceptable health status UNAVAILABLE

I20240502 21:35:08.506250 11552 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240502 21:29:07.892156 11556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:07.892833 11556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:07.910315 11556 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:29:12.174316 11556 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 21:29:14.000150 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 445d2e6ec7154b80abec5fff346b8f5e: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:14.005887 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5802da0404e74a07bca290cdc59b279b: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:14.011374 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef412fa8fe9249729cb99879a5c8343a: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:14.011725 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f52c6c3804274777abfa00bb33ae9b16: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:14.011924 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c0a0ef39a2a246449eb582cc1d993484: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:14.012212 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e8eef05f850243d0bdc7e1899f8c6139: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:14.012389 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 265d5656d7c048aa9795a906661b8ea8: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:14.012547 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16e8100140034581b2a773dcf740998d: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:14.012842 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1b17408dbab9492bba0ed50efd6ef43f: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:14.013020 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a237d7bda970485480e98020c6ac738f: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:14.013305 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f52c6c3804274777abfa00bb33ae9b16: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:14.013512 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e8eef05f850243d0bdc7e1899f8c6139: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:14.013650 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 265d5656d7c048aa9795a906661b8ea8: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:14.013978 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16e8100140034581b2a773dcf740998d: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server e0de0545fa0547d3929111e990e6d1b2
I20240502 21:29:14.326922 11556 rebalancer_tool.cc:1084] tablet 4d1a31513bb64578a85adf1f622e5bc0: '525a093ea5bf40dc9bbf9dc39895d670' -> '6ef0beaf2f6941afb6ea9f9f274517fe' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240502 21:29:14.327245 11556 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 21:29:16.638960 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f52c6c3804274777abfa00bb33ae9b16: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:16.639305 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c0a0ef39a2a246449eb582cc1d993484: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:16.646785 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f52c6c3804274777abfa00bb33ae9b16: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:16.647053 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16e8100140034581b2a773dcf740998d: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:16.647171 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c0a0ef39a2a246449eb582cc1d993484: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:16.647274 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef412fa8fe9249729cb99879a5c8343a: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:16.647353 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 445d2e6ec7154b80abec5fff346b8f5e: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:16.647511 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a237d7bda970485480e98020c6ac738f: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:16.647799 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 265d5656d7c048aa9795a906661b8ea8: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:16.647946 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e8eef05f850243d0bdc7e1899f8c6139: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:16.648166 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1b17408dbab9492bba0ed50efd6ef43f: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:16.648260 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a237d7bda970485480e98020c6ac738f: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:16.648355 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5802da0404e74a07bca290cdc59b279b: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:16.648460 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f52c6c3804274777abfa00bb33ae9b16: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server e0de0545fa0547d3929111e990e6d1b2
W20240502 21:29:16.648533 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c0a0ef39a2a246449eb582cc1d993484: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:16.648633 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e8eef05f850243d0bdc7e1899f8c6139: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:16.648725 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16e8100140034581b2a773dcf740998d: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server e0de0545fa0547d3929111e990e6d1b2
I20240502 21:29:16.749521 11556 rebalancer_tool.cc:1074] tablet 4d1a31513bb64578a85adf1f622e5bc0: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move scheduled
I20240502 21:29:17.509943 11556 rebalancer_tool.cc:1074] tablet a7c6b881c9ef4cdca787f065778e526f: '5543ddab5b484642b139b7c848932a58' -> 'e0de0545fa0547d3929111e990e6d1b2' move scheduled
I20240502 21:29:18.148141 11556 rebalancer_tool.cc:1074] tablet 87f35aff2b104c01a9d9d48a7e9c9b23: '19f162e164854f2da623a4bb8f9e6bf6' -> '4cbb5b06a6a44eb6b2145cd538b01af2' move scheduled
I20240502 21:29:18.932070 11556 rebalancer_tool.cc:1074] tablet de3287296c4547688388388dd4c7e638: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> 'd10e9e156ba74a97a221c85b68ab8bc4' move scheduled
I20240502 21:29:19.474056 11556 rebalancer_tool.cc:1074] tablet f82c35077224414cb284472d49d7de4f: '525a093ea5bf40dc9bbf9dc39895d670' -> '6ef0beaf2f6941afb6ea9f9f274517fe' move scheduled
I20240502 21:29:20.098259 11556 rebalancer_tool.cc:1074] tablet 41ee481a3ece47a0905e93c66dbf949d: '5543ddab5b484642b139b7c848932a58' -> 'e0de0545fa0547d3929111e990e6d1b2' move scheduled
I20240502 21:29:20.825858 11556 rebalancer_tool.cc:1074] tablet 32c306d724954a16be21a34aa4cc0b5f: '19f162e164854f2da623a4bb8f9e6bf6' -> '4cbb5b06a6a44eb6b2145cd538b01af2' move scheduled
I20240502 21:29:21.609467 11556 rebalancer_tool.cc:1074] tablet f9ae8fac693d4d68b6741f2509e1265e: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> 'd10e9e156ba74a97a221c85b68ab8bc4' move scheduled
I20240502 21:29:22.026252 11556 rebalancer_tool.cc:1074] tablet 3caa36212ba444aa88d153ff9f90122e: '19f162e164854f2da623a4bb8f9e6bf6' -> '6ef0beaf2f6941afb6ea9f9f274517fe' move scheduled
I20240502 21:29:22.408049 11556 rebalancer_tool.cc:1084] tablet 94d2cdf8ae6941668b5f680c558656e8: '525a093ea5bf40dc9bbf9dc39895d670' -> 'e0de0545fa0547d3929111e990e6d1b2' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240502 21:29:22.408253 11556 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 21:29:23.532058 13787 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.27.196.126:34625, user_credentials={real_user=slave}} blocked reactor thread for 111469us
W20240502 21:29:24.494377 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 265d5656d7c048aa9795a906661b8ea8: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:24.515950 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f52c6c3804274777abfa00bb33ae9b16: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:24.516211 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef412fa8fe9249729cb99879a5c8343a: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:24.516395 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e8eef05f850243d0bdc7e1899f8c6139: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:24.516646 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16e8100140034581b2a773dcf740998d: could not find any suitable replica to move from server 19f162e164854f2da623a4bb8f9e6bf6 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:24.516796 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5802da0404e74a07bca290cdc59b279b: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server 6ef0beaf2f6941afb6ea9f9f274517fe
W20240502 21:29:24.517102 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f52c6c3804274777abfa00bb33ae9b16: could not find any suitable replica to move from server f2fdd596a8b44bbc848fe40f9f07fbe0 to server d10e9e156ba74a97a221c85b68ab8bc4
W20240502 21:29:24.517278 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 265d5656d7c048aa9795a906661b8ea8: could not find any suitable replica to move from server 5543ddab5b484642b139b7c848932a58 to server 4cbb5b06a6a44eb6b2145cd538b01af2
W20240502 21:29:24.517390 11556 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 445d2e6ec7154b80abec5fff346b8f5e: could not find any suitable replica to move from server 525a093ea5bf40dc9bbf9dc39895d670 to server 6ef0beaf2f6941afb6ea9f9f274517fe
I20240502 21:29:24.997038 11556 rebalancer_tool.cc:1074] tablet 58c7004ceb044f5fa9dec54910046b69: '6ef0beaf2f6941afb6ea9f9f274517fe' -> '' move scheduled
I20240502 21:29:25.219705 11556 rebalancer_tool.cc:1074] tablet aeee59c5512d448d8f986f959dce1cc8: '5543ddab5b484642b139b7c848932a58' -> '' move scheduled
I20240502 21:29:25.349999 11556 rebalancer_tool.cc:1074] tablet 3d1a5050d3f944969a560994a23700ad: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move scheduled
I20240502 21:29:25.447165 11556 rebalancer_tool.cc:1074] tablet 1497b781f36c43b39d22a05bb524c59d: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '' move scheduled
I20240502 21:29:33.848906 11556 rebalancer_tool.cc:1136] tablet 58c7004ceb044f5fa9dec54910046b69: '6ef0beaf2f6941afb6ea9f9f274517fe' -> '' move completed: OK
W20240502 21:29:35.422880 15315 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.27.196.126:34625, user_credentials={real_user=slave}} blocked reactor thread for 56056.4us
I20240502 21:29:41.554899 11556 rebalancer_tool.cc:1136] tablet a7c6b881c9ef4cdca787f065778e526f: '5543ddab5b484642b139b7c848932a58' -> 'e0de0545fa0547d3929111e990e6d1b2' move completed: OK
I20240502 21:29:54.987450 11556 rebalancer_tool.cc:1136] tablet 87f35aff2b104c01a9d9d48a7e9c9b23: '19f162e164854f2da623a4bb8f9e6bf6' -> '4cbb5b06a6a44eb6b2145cd538b01af2' move completed: OK
I20240502 21:29:58.154778 11556 rebalancer_tool.cc:1074] tablet cffcb4f7f7754262beeb06168097f6dc: '5543ddab5b484642b139b7c848932a58' -> '' move scheduled
I20240502 21:29:58.253082 11556 rebalancer_tool.cc:1136] tablet cffcb4f7f7754262beeb06168097f6dc: '5543ddab5b484642b139b7c848932a58' -> '' move completed: OK
I20240502 21:29:58.344967 11556 rebalancer_tool.cc:1136] tablet 3d1a5050d3f944969a560994a23700ad: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move completed: OK
I20240502 21:29:58.532339 11556 rebalancer_tool.cc:1136] tablet 1497b781f36c43b39d22a05bb524c59d: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '' move completed: OK
I20240502 21:29:58.659987 11556 rebalancer_tool.cc:1136] tablet aeee59c5512d448d8f986f959dce1cc8: '5543ddab5b484642b139b7c848932a58' -> '' move completed: OK
I20240502 21:29:58.922452 11556 rebalancer_tool.cc:1136] tablet f9ae8fac693d4d68b6741f2509e1265e: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> 'd10e9e156ba74a97a221c85b68ab8bc4' move completed: OK
I20240502 21:29:59.178119 11556 rebalancer_tool.cc:1136] tablet 32c306d724954a16be21a34aa4cc0b5f: '19f162e164854f2da623a4bb8f9e6bf6' -> '4cbb5b06a6a44eb6b2145cd538b01af2' move completed: OK
I20240502 21:29:59.310310 11556 rebalancer_tool.cc:1136] tablet 41ee481a3ece47a0905e93c66dbf949d: '5543ddab5b484642b139b7c848932a58' -> 'e0de0545fa0547d3929111e990e6d1b2' move completed: OK
I20240502 21:29:59.503749 11556 rebalancer_tool.cc:1136] tablet 4d1a31513bb64578a85adf1f622e5bc0: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move completed: OK
I20240502 21:29:59.599632 11556 rebalancer_tool.cc:1136] tablet de3287296c4547688388388dd4c7e638: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> 'd10e9e156ba74a97a221c85b68ab8bc4' move completed: OK
I20240502 21:29:59.758342 11556 rebalancer_tool.cc:1136] tablet 3caa36212ba444aa88d153ff9f90122e: '19f162e164854f2da623a4bb8f9e6bf6' -> '6ef0beaf2f6941afb6ea9f9f274517fe' move completed: OK
I20240502 21:29:59.854352 11556 rebalancer_tool.cc:1136] tablet f82c35077224414cb284472d49d7de4f: '525a093ea5bf40dc9bbf9dc39895d670' -> '6ef0beaf2f6941afb6ea9f9f274517fe' move completed: OK
I20240502 21:30:00.067927 11556 rebalancer_tool.cc:1074] tablet a2f3edaf1c874cefb820f027f90d1185: '5543ddab5b484642b139b7c848932a58' -> '' move scheduled
I20240502 21:30:00.130932 11556 rebalancer_tool.cc:1074] tablet be34ec89876d42b69efe1878c0d9bc19: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '' move scheduled
I20240502 21:30:00.204013 11556 rebalancer_tool.cc:1074] tablet cdf72d25f0f847e5998acbc53979e64d: '19f162e164854f2da623a4bb8f9e6bf6' -> '' move scheduled
I20240502 21:30:00.317713 11556 rebalancer_tool.cc:1074] tablet dc1970b9263d41abb575b9d8639c59ae: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move scheduled
I20240502 21:30:00.506019 11556 rebalancer_tool.cc:1074] tablet 4674e376bf7440cda619784491685b44: '5543ddab5b484642b139b7c848932a58' -> '' move scheduled
W20240502 21:30:01.517747 19004 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.27.196.126:34625, user_credentials={real_user=slave}} blocked reactor thread for 59142.8us
I20240502 21:30:02.630473 11556 rebalancer_tool.cc:1136] tablet dc1970b9263d41abb575b9d8639c59ae: '525a093ea5bf40dc9bbf9dc39895d670' -> '' move completed: OK
I20240502 21:30:02.805701 11556 rebalancer_tool.cc:1136] tablet cdf72d25f0f847e5998acbc53979e64d: '19f162e164854f2da623a4bb8f9e6bf6' -> '' move completed: OK
I20240502 21:30:02.955255 11556 rebalancer_tool.cc:1136] tablet be34ec89876d42b69efe1878c0d9bc19: 'f2fdd596a8b44bbc848fe40f9f07fbe0' -> '' move completed: OK
I20240502 21:30:03.081311 11556 rebalancer_tool.cc:1136] tablet a2f3edaf1c874cefb820f027f90d1185: '5543ddab5b484642b139b7c848932a58' -> '' move completed: OK
I20240502 21:30:03.256991 11556 rebalancer_tool.cc:1074] tablet 94d2cdf8ae6941668b5f680c558656e8: '19f162e164854f2da623a4bb8f9e6bf6' -> '' move scheduled
I20240502 21:30:03.410284 11556 rebalancer_tool.cc:1074] tablet 9240163cfac446d2a14dbd9fcbbf9ff7: '19f162e164854f2da623a4bb8f9e6bf6' -> '' move scheduled
I20240502 21:30:03.596335 11556 rebalancer_tool.cc:1074] tablet 51936fb1f678458a8795f8246590addb: '19f162e164854f2da623a4bb8f9e6bf6' -> '' move scheduled
I20240502 21:30:03.789325 11556 rebalancer_tool.cc:1136] tablet 51936fb1f678458a8795f8246590addb: '19f162e164854f2da623a4bb8f9e6bf6' -> '' move completed: OK
W20240502 21:30:05.380282 19639 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 21:30:05.157675 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.196.70:44211 (local address 127.0.0.1:34804)
0502 21:30:05.158280 (+   605us) negotiation.cc:107] Waiting for socket to connect
0502 21:30:05.158307 (+    27us) client_negotiation.cc:174] Beginning negotiation
0502 21:30:05.158502 (+   195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 21:30:05.378318 (+219816us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":452}
I20240502 21:30:06.218937 11556 rebalancer_tool.cc:1136] tablet 94d2cdf8ae6941668b5f680c558656e8: '19f162e164854f2da623a4bb8f9e6bf6' -> '' move completed: OK
I20240502 21:30:06.349545 11556 rebalancer_tool.cc:1136] tablet 4674e376bf7440cda619784491685b44: '5543ddab5b484642b139b7c848932a58' -> '' move completed: OK
I20240502 21:35:06.907835 11556 rebalancer_tool.cc:1136] tablet 9240163cfac446d2a14dbd9fcbbf9ff7: '19f162e164854f2da623a4bb8f9e6bf6' -> '' move completed: OK
Illegal state: tablet server 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): unacceptable health status UNAVAILABLE

E20240502 21:35:08.507349 28433 external_mini_cluster.cc:943] Process with UUID 6ef0beaf2f6941afb6ea9f9f274517fe has crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
  0
  num_crashes
    Which is: 1
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
  Actual: it does.
I20240502 21:35:08.509219 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 9023
I20240502 21:35:08.517789  9148 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:35:08.991585 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 9023
W20240502 21:35:09.023371  8946 catalog_manager.cc:4573] TS 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet cdf72d25f0f847e5998acbc53979e64d: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:35:09.060143 11027 connection.cc:535] server connection from 127.27.196.65:42255 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:09.060473 11295 connection.cc:535] server connection from 127.27.196.65:48577 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:09.060705 11431 connection.cc:535] server connection from 127.27.196.65:35315 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:09.060976  9172 connection.cc:535] client connection to 127.27.196.65:42177 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:35:09.061129 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 9157
W20240502 21:35:09.061138  9444 connection.cc:535] client connection to 127.27.196.65:42177 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:09.061269  9444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 14 similar messages]
W20240502 21:35:09.071081  9172 connection.cc:535] server connection from 127.27.196.65:48383 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:09.075014  9444 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Couldn't send request to peer 5543ddab5b484642b139b7c848932a58. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:35:09.075191  9444 consensus_peers.cc:485] T be34ec89876d42b69efe1878c0d9bc19 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Couldn't send request to peer 5543ddab5b484642b139b7c848932a58. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:35:09.075279  9444 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Couldn't send request to peer 5543ddab5b484642b139b7c848932a58. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:35:09.075619  9172 consensus_peers.cc:485] T 3caa36212ba444aa88d153ff9f90122e P 525a093ea5bf40dc9bbf9dc39895d670 -> Peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Couldn't send request to peer 5543ddab5b484642b139b7c848932a58. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:09.080994  9282 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:35:09.142300  9310 consensus_peers.cc:485] T 4674e376bf7440cda619784491685b44 P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Couldn't send request to peer 5543ddab5b484642b139b7c848932a58. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:35:09.254354 11429 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Couldn't send request to peer 5543ddab5b484642b139b7c848932a58. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:09.506129 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 9157
W20240502 21:35:09.576705 11431 connection.cc:535] server connection from 127.27.196.66:44945 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:09.577435  9313 connection.cc:535] client connection to 127.27.196.66:39071 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:35:09.577636 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 9293
W20240502 21:35:09.577643  9444 connection.cc:535] server connection from 127.27.196.66:58097 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:09.578287  9447 consensus_peers.cc:485] T c56ac926025e47218e9df4088184a278 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Couldn't send request to peer 525a093ea5bf40dc9bbf9dc39895d670. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:35:09.579393  9313 consensus_peers.cc:485] T cffcb4f7f7754262beeb06168097f6dc P f2fdd596a8b44bbc848fe40f9f07fbe0 -> Peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Couldn't send request to peer 525a093ea5bf40dc9bbf9dc39895d670. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:35:09.579629 11432 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240502 21:35:09.580541 11432 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Couldn't send request to peer 525a093ea5bf40dc9bbf9dc39895d670. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:09.585151  9419 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:35:09.632140 11298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111)
W20240502 21:35:09.634056 11298 consensus_peers.cc:485] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 -> Peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Couldn't send request to peer 525a093ea5bf40dc9bbf9dc39895d670. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:09.646793 24439 raft_consensus.cc:482] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:09.647092 24439 raft_consensus.cc:504] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:09.648105 24439 leader_election.cc:288] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:35:09.650122 11295 leader_election.cc:334] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:09.655943 11093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9240163cfac446d2a14dbd9fcbbf9ff7" candidate_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" is_pre_election: true
I20240502 21:35:09.656720 11296 leader_election.cc:302] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cbb5b06a6a44eb6b2145cd538b01af2; no voters: 5543ddab5b484642b139b7c848932a58, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:35:09.657083 24439 raft_consensus.cc:2726] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:35:09.762050 24840 raft_consensus.cc:482] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:09.762313 24840 raft_consensus.cc:504] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:35:09.763295 24840 leader_election.cc:288] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857)
I20240502 21:35:09.764715 11365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c306d724954a16be21a34aa4cc0b5f" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" is_pre_election: true
W20240502 21:35:09.764971  9444 leader_election.cc:334] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:09.765874  9444 leader_election.cc:302] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6; no voters: 4cbb5b06a6a44eb6b2145cd538b01af2, 5543ddab5b484642b139b7c848932a58
I20240502 21:35:09.766228 24840 raft_consensus.cc:2726] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:35:09.810922 11028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111)
W20240502 21:35:09.813287 11028 consensus_peers.cc:485] T f9ae8fac693d4d68b6741f2509e1265e P d10e9e156ba74a97a221c85b68ab8bc4 -> Peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Couldn't send request to peer 525a093ea5bf40dc9bbf9dc39895d670. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:09.842016 24840 raft_consensus.cc:482] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:09.842258 24840 raft_consensus.cc:504] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:09.844141 11365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b4486f03fd9474aa0bb2cb4c9149437" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" is_pre_election: true
I20240502 21:35:09.844358 11093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b4486f03fd9474aa0bb2cb4c9149437" candidate_uuid: "19f162e164854f2da623a4bb8f9e6bf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" is_pre_election: true
I20240502 21:35:09.844851 24840 leader_election.cc:288] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:35:09.844813  9444 leader_election.cc:334] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:09.845083  9445 leader_election.cc:302] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6; no voters: 5543ddab5b484642b139b7c848932a58, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:35:09.845468 24840 raft_consensus.cc:2726] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:35:09.861451 24439 raft_consensus.cc:482] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:09.861711 24439 raft_consensus.cc:504] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:35:09.862659 24439 leader_election.cc:288] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
I20240502 21:35:09.863636  9508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c306d724954a16be21a34aa4cc0b5f" candidate_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6" is_pre_election: true
I20240502 21:35:09.863916  9508 raft_consensus.cc:2454] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cbb5b06a6a44eb6b2145cd538b01af2 in term 1.
W20240502 21:35:09.864389 11295 leader_election.cc:334] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:09.864643 11295 leader_election.cc:302] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 4cbb5b06a6a44eb6b2145cd538b01af2; no voters: 5543ddab5b484642b139b7c848932a58
I20240502 21:35:09.864997 24439 raft_consensus.cc:2781] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:35:09.865100 24439 raft_consensus.cc:482] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:09.865199 24439 raft_consensus.cc:3037] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:35:09.867956 24439 raft_consensus.cc:504] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:35:09.868731 24439 leader_election.cc:288] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 election: Requested vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
I20240502 21:35:09.869786  9508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c306d724954a16be21a34aa4cc0b5f" candidate_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6"
I20240502 21:35:09.870394  9508 raft_consensus.cc:3037] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:35:09.870678 11295 leader_election.cc:334] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:09.874258  9508 raft_consensus.cc:2454] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cbb5b06a6a44eb6b2145cd538b01af2 in term 2.
I20240502 21:35:09.874787 11295 leader_election.cc:302] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6, 4cbb5b06a6a44eb6b2145cd538b01af2; no voters: 5543ddab5b484642b139b7c848932a58
I20240502 21:35:09.875701 24439 raft_consensus.cc:2781] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:35:09.875901 24439 raft_consensus.cc:686] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 LEADER]: Becoming Leader. State: Replica: 4cbb5b06a6a44eb6b2145cd538b01af2, State: Running, Role: LEADER
I20240502 21:35:09.876375 24439 consensus_queue.cc:237] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } }
I20240502 21:35:09.884810  8955 catalog_manager.cc:5484] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 reported cstate change: term changed from 1 to 2, leader changed from 5543ddab5b484642b139b7c848932a58 (127.27.196.65) to 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71). New cstate: current_term: 2 leader_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } }
I20240502 21:35:09.886523 24840 raft_consensus.cc:482] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:09.886775 24840 raft_consensus.cc:504] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } }
I20240502 21:35:09.887691 24840 leader_election.cc:288] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
W20240502 21:35:09.889883  9444 leader_election.cc:334] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:09.933768 24960 raft_consensus.cc:482] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:09.933990 24960 raft_consensus.cc:504] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } }
I20240502 21:35:09.934746 24960 leader_election.cc:288] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
W20240502 21:35:09.936226 11298 leader_election.cc:334] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111)
W20240502 21:35:09.936391 11295 leader_election.cc:334] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:09.936504 11295 leader_election.cc:302] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cbb5b06a6a44eb6b2145cd538b01af2; no voters: 525a093ea5bf40dc9bbf9dc39895d670, 5543ddab5b484642b139b7c848932a58
I20240502 21:35:09.936774 24960 raft_consensus.cc:2726] T cdf72d25f0f847e5998acbc53979e64d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:35:10.011369 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 9293
W20240502 21:35:10.082175 11298 connection.cc:535] server connection from 127.27.196.67:43725 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:10.082196 11432 connection.cc:535] server connection from 127.27.196.67:39199 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:10.082201 11026 connection.cc:535] server connection from 127.27.196.67:54351 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:10.082702  9446 connection.cc:535] client connection to 127.27.196.67:43203 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:10.082769 11431 connection.cc:535] client connection to 127.27.196.67:43203 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:35:10.083046 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 9429
W20240502 21:35:10.084250 11431 consensus_peers.cc:485] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Couldn't send request to peer f2fdd596a8b44bbc848fe40f9f07fbe0. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.67:43203: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:35:10.084228  9446 leader_election.cc:334] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Network error: Client connection negotiation failed: client connection to 127.27.196.67:43203: connect: Connection refused (error 111)
I20240502 21:35:10.084426  9446 leader_election.cc:302] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19f162e164854f2da623a4bb8f9e6bf6; no voters: 5543ddab5b484642b139b7c848932a58, f2fdd596a8b44bbc848fe40f9f07fbe0
I20240502 21:35:10.084792 24840 raft_consensus.cc:2726] T dc1970b9263d41abb575b9d8639c59ae P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:35:10.084934  9446 consensus_peers.cc:485] T 3d1a5050d3f944969a560994a23700ad P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Couldn't send request to peer f2fdd596a8b44bbc848fe40f9f07fbe0. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.67:43203: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:35:10.085076  9446 consensus_peers.cc:485] T a2f3edaf1c874cefb820f027f90d1185 P 19f162e164854f2da623a4bb8f9e6bf6 -> Peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Couldn't send request to peer f2fdd596a8b44bbc848fe40f9f07fbe0. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.67:43203: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:10.090308  9554 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:35:10.160821 24539 raft_consensus.cc:482] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 525a093ea5bf40dc9bbf9dc39895d670)
I20240502 21:35:10.161068 24539 raft_consensus.cc:504] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:35:10.162042 24539 leader_election.cc:288] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
I20240502 21:35:10.162941 11090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51936fb1f678458a8795f8246590addb" candidate_uuid: "e0de0545fa0547d3929111e990e6d1b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" is_pre_election: true
W20240502 21:35:10.164436 11432 leader_election.cc:334] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111)
I20240502 21:35:10.164623 11432 leader_election.cc:302] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 525a093ea5bf40dc9bbf9dc39895d670, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:35:10.164920 24539 raft_consensus.cc:2726] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:35:10.402683 11295 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 -> Peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Couldn't send request to peer 5543ddab5b484642b139b7c848932a58. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:10.482177 24960 raft_consensus.cc:482] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:10.482432 24960 raft_consensus.cc:504] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:10.483389 24960 leader_election.cc:288] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
I20240502 21:35:10.484033 11090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b4486f03fd9474aa0bb2cb4c9149437" candidate_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" is_pre_election: true
W20240502 21:35:10.485229 11295 leader_election.cc:334] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:10.485353 11295 leader_election.cc:302] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4cbb5b06a6a44eb6b2145cd538b01af2; no voters: 5543ddab5b484642b139b7c848932a58, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:35:10.485646 24960 raft_consensus.cc:2726] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:35:10.490793 22998 raft_consensus.cc:482] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:10.491149 22998 raft_consensus.cc:504] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:10.492874 24539 raft_consensus.cc:482] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:10.493093 24539 raft_consensus.cc:504] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:10.493206 22998 leader_election.cc:288] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467)
I20240502 21:35:10.493376 22998 raft_consensus.cc:482] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:10.493870 24539 leader_election.cc:288] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
I20240502 21:35:10.494027 24539 raft_consensus.cc:482] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:10.494166 24539 raft_consensus.cc:504] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:10.494948 24539 leader_election.cc:288] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:35:10.498492 11431 leader_election.cc:334] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Network error: Client connection negotiation failed: client connection to 127.27.196.67:43203: connect: Connection refused (error 111)
W20240502 21:35:10.499827 11027 leader_election.cc:334] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Network error: Client connection negotiation failed: client connection to 127.27.196.67:43203: connect: Connection refused (error 111)
W20240502 21:35:10.503964 11429 leader_election.cc:334] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
W20240502 21:35:10.504124 11429 leader_election.cc:334] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:10.504221 11429 leader_election.cc:302] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 5543ddab5b484642b139b7c848932a58, f2fdd596a8b44bbc848fe40f9f07fbe0
I20240502 21:35:10.504525 24974 raft_consensus.cc:2726] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:35:10.505589  9508 raft_consensus.cc:1261] T 32c306d724954a16be21a34aa4cc0b5f P 19f162e164854f2da623a4bb8f9e6bf6 [term 2 FOLLOWER]: Refusing update from remote peer 4cbb5b06a6a44eb6b2145cd538b01af2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 21:35:10.506424  9507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b4486f03fd9474aa0bb2cb4c9149437" candidate_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "19f162e164854f2da623a4bb8f9e6bf6" is_pre_election: true
I20240502 21:35:10.506721  9507 raft_consensus.cc:2454] T 8b4486f03fd9474aa0bb2cb4c9149437 P 19f162e164854f2da623a4bb8f9e6bf6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cbb5b06a6a44eb6b2145cd538b01af2 in term 1.
I20240502 21:35:10.507485 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 9429
I20240502 21:35:10.508826 22998 raft_consensus.cc:504] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:10.510265 22998 leader_election.cc:288] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857)
I20240502 21:35:10.510499 22998 raft_consensus.cc:482] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:10.510681 22998 raft_consensus.cc:504] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:10.511534 22998 leader_election.cc:288] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467)
I20240502 21:35:10.511972 22998 raft_consensus.cc:482] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:10.512230 22998 raft_consensus.cc:504] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:10.513098 22998 leader_election.cc:288] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071)
I20240502 21:35:10.513913 24977 raft_consensus.cc:482] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:10.514159 24977 raft_consensus.cc:504] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:10.514500 11493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dedaa327b6479e8325f91481c1688e" candidate_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "e0de0545fa0547d3929111e990e6d1b2" is_pre_election: true
I20240502 21:35:10.514559 11495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1a31513bb64578a85adf1f622e5bc0" candidate_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: false dest_uuid: "e0de0545fa0547d3929111e990e6d1b2" is_pre_election: true
I20240502 21:35:10.514806 11493 raft_consensus.cc:2454] T 79dedaa327b6479e8325f91481c1688e P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d10e9e156ba74a97a221c85b68ab8bc4 in term 1.
I20240502 21:35:10.514873 11495 raft_consensus.cc:2454] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d10e9e156ba74a97a221c85b68ab8bc4 in term 3.
I20240502 21:35:10.515022 24978 raft_consensus.cc:482] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 525a093ea5bf40dc9bbf9dc39895d670)
I20240502 21:35:10.515195 11363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b4486f03fd9474aa0bb2cb4c9149437" candidate_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" is_pre_election: true
I20240502 21:35:10.515198 24978 raft_consensus.cc:504] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:35:10.515455 11363 raft_consensus.cc:2454] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d10e9e156ba74a97a221c85b68ab8bc4 in term 1.
I20240502 21:35:10.516165 24977 leader_election.cc:288] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857)
I20240502 21:35:10.516266 11090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1a31513bb64578a85adf1f622e5bc0" candidate_uuid: "e0de0545fa0547d3929111e990e6d1b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: false dest_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" is_pre_election: true
I20240502 21:35:10.516175 24978 leader_election.cc:288] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:35:10.516511 11090 raft_consensus.cc:2454] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0de0545fa0547d3929111e990e6d1b2 in term 3.
I20240502 21:35:10.517030 11090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dedaa327b6479e8325f91481c1688e" candidate_uuid: "e0de0545fa0547d3929111e990e6d1b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" is_pre_election: true
I20240502 21:35:10.517235 11090 raft_consensus.cc:2454] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0de0545fa0547d3929111e990e6d1b2 in term 1.
I20240502 21:35:10.517726 11363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9240163cfac446d2a14dbd9fcbbf9ff7" candidate_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" is_pre_election: true
W20240502 21:35:10.517827 11025 leader_election.cc:334] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
W20240502 21:35:10.517992 11025 leader_election.cc:334] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:10.517980 11363 raft_consensus.cc:2454] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d10e9e156ba74a97a221c85b68ab8bc4 in term 1.
I20240502 21:35:10.518106 11025 leader_election.cc:302] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d10e9e156ba74a97a221c85b68ab8bc4; no voters: 5543ddab5b484642b139b7c848932a58, f2fdd596a8b44bbc848fe40f9f07fbe0
I20240502 21:35:10.518499 11430 leader_election.cc:302] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d10e9e156ba74a97a221c85b68ab8bc4, e0de0545fa0547d3929111e990e6d1b2; no voters: 5543ddab5b484642b139b7c848932a58
W20240502 21:35:10.518616 11025 leader_election.cc:334] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:10.518654 24978 raft_consensus.cc:2726] T 79dedaa327b6479e8325f91481c1688e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:35:10.518775 11025 leader_election.cc:334] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:10.519235 11025 leader_election.cc:302] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cbb5b06a6a44eb6b2145cd538b01af2, d10e9e156ba74a97a221c85b68ab8bc4; no voters: 
I20240502 21:35:10.519470 24974 raft_consensus.cc:2781] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 21:35:10.519526 24978 raft_consensus.cc:2781] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:35:10.519591 24974 raft_consensus.cc:482] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:10.519614 24978 raft_consensus.cc:482] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:10.519686 24974 raft_consensus.cc:3037] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:35:10.519728 24978 raft_consensus.cc:3037] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:35:10.522567 24974 raft_consensus.cc:504] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:10.523880 24974 leader_election.cc:288] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 election: Requested vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
I20240502 21:35:10.523751 24978 raft_consensus.cc:504] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:10.524443 11028 leader_election.cc:302] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d10e9e156ba74a97a221c85b68ab8bc4, e0de0545fa0547d3929111e990e6d1b2; no voters: 5543ddab5b484642b139b7c848932a58
I20240502 21:35:10.524616 24978 leader_election.cc:288] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 election: Requested vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857)
I20240502 21:35:10.524930 24978 raft_consensus.cc:2781] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240502 21:35:10.525058 11028 leader_election.cc:334] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111)
I20240502 21:35:10.525314 11028 leader_election.cc:302] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d10e9e156ba74a97a221c85b68ab8bc4; no voters: 525a093ea5bf40dc9bbf9dc39895d670, 5543ddab5b484642b139b7c848932a58
I20240502 21:35:10.525949 11363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9240163cfac446d2a14dbd9fcbbf9ff7" candidate_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2"
I20240502 21:35:10.526206 11363 raft_consensus.cc:3037] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:35:10.526608 11025 leader_election.cc:334] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
W20240502 21:35:10.526891 11025 leader_election.cc:334] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
W20240502 21:35:10.527724 11025 leader_election.cc:334] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:10.528062 24977 raft_consensus.cc:2726] T e46065d94e874293a836bc74241b8e06 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:35:10.528162 11429 leader_election.cc:334] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:10.525095 24978 raft_consensus.cc:482] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5543ddab5b484642b139b7c848932a58)
I20240502 21:35:10.528527 24978 raft_consensus.cc:3037] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:35:10.529630 11090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1a31513bb64578a85adf1f622e5bc0" candidate_uuid: "e0de0545fa0547d3929111e990e6d1b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: false dest_uuid: "d10e9e156ba74a97a221c85b68ab8bc4"
I20240502 21:35:10.530279 11363 raft_consensus.cc:2454] T 9240163cfac446d2a14dbd9fcbbf9ff7 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d10e9e156ba74a97a221c85b68ab8bc4 in term 2.
I20240502 21:35:10.530879 11025 leader_election.cc:302] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cbb5b06a6a44eb6b2145cd538b01af2, d10e9e156ba74a97a221c85b68ab8bc4; no voters: 5543ddab5b484642b139b7c848932a58
I20240502 21:35:10.531334 24977 raft_consensus.cc:2781] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:35:10.531517 24977 raft_consensus.cc:686] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [term 2 LEADER]: Becoming Leader. State: Replica: d10e9e156ba74a97a221c85b68ab8bc4, State: Running, Role: LEADER
I20240502 21:35:10.531893 24977 consensus_queue.cc:237] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
W20240502 21:35:10.532020 11028 leader_election.cc:334] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111)
I20240502 21:35:10.532395 11028 leader_election.cc:302] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d10e9e156ba74a97a221c85b68ab8bc4; no voters: 525a093ea5bf40dc9bbf9dc39895d670, 5543ddab5b484642b139b7c848932a58
I20240502 21:35:10.532575 24978 raft_consensus.cc:504] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } }
I20240502 21:35:10.532902 24976 raft_consensus.cc:2726] T 3caa36212ba444aa88d153ff9f90122e P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:35:10.533103 11090 raft_consensus.cc:2379] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e0de0545fa0547d3929111e990e6d1b2 in current term 4: Already voted for candidate d10e9e156ba74a97a221c85b68ab8bc4 in this term.
I20240502 21:35:10.534294 11430 leader_election.cc:302] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 5543ddab5b484642b139b7c848932a58, d10e9e156ba74a97a221c85b68ab8bc4
I20240502 21:35:10.535576 24978 leader_election.cc:288] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 4 election: Requested vote from peers 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177), e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467)
I20240502 21:35:10.535831 11493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1a31513bb64578a85adf1f622e5bc0" candidate_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: false dest_uuid: "e0de0545fa0547d3929111e990e6d1b2"
I20240502 21:35:10.536392 11493 raft_consensus.cc:2379] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d10e9e156ba74a97a221c85b68ab8bc4 in current term 4: Already voted for candidate e0de0545fa0547d3929111e990e6d1b2 in this term.
W20240502 21:35:10.536592 11025 leader_election.cc:334] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:10.536522  8955 catalog_manager.cc:5484] T 9240163cfac446d2a14dbd9fcbbf9ff7 P d10e9e156ba74a97a221c85b68ab8bc4 reported cstate change: term changed from 1 to 2, leader changed from 5543ddab5b484642b139b7c848932a58 (127.27.196.65) to d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69). New cstate: current_term: 2 leader_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } }
I20240502 21:35:10.537031 24974 raft_consensus.cc:2726] T 4d1a31513bb64578a85adf1f622e5bc0 P e0de0545fa0547d3929111e990e6d1b2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:35:10.537170 11028 leader_election.cc:302] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d10e9e156ba74a97a221c85b68ab8bc4; no voters: 5543ddab5b484642b139b7c848932a58, e0de0545fa0547d3929111e990e6d1b2
I20240502 21:35:10.537658 24977 raft_consensus.cc:2726] T 4d1a31513bb64578a85adf1f622e5bc0 P d10e9e156ba74a97a221c85b68ab8bc4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240502 21:35:10.593297 11430 connection.cc:535] server connection from 127.27.196.68:36505 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:10.594033 11025 connection.cc:535] client connection to 127.27.196.68:40931 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:35:10.594092 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 11009
W20240502 21:35:10.595700 11295 leader_election.cc:334] T 8b4486f03fd9474aa0bb2cb4c9149437 P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): Network error: Client connection negotiation failed: client connection to 127.27.196.68:40931: connect: Connection refused (error 111)
W20240502 21:35:10.596032 11025 leader_election.cc:334] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): Network error: Client connection negotiation failed: client connection to 127.27.196.68:40931: connect: Connection refused (error 111)
I20240502 21:35:10.596201 11025 leader_election.cc:302] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4cbb5b06a6a44eb6b2145cd538b01af2, d10e9e156ba74a97a221c85b68ab8bc4; no voters: 19f162e164854f2da623a4bb8f9e6bf6, 5543ddab5b484642b139b7c848932a58
W20240502 21:35:10.596174 11295 consensus_peers.cc:485] T f0259f01acb249dd99c662151dc21916 P 4cbb5b06a6a44eb6b2145cd538b01af2 -> Peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): Couldn't send request to peer 19f162e164854f2da623a4bb8f9e6bf6. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.68:40931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:35:10.596415 11295 consensus_peers.cc:485] T 32c306d724954a16be21a34aa4cc0b5f P 4cbb5b06a6a44eb6b2145cd538b01af2 -> Peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): Couldn't send request to peer 19f162e164854f2da623a4bb8f9e6bf6. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.68:40931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:10.596693 24977 raft_consensus.cc:2726] T 8b4486f03fd9474aa0bb2cb4c9149437 P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:35:10.607625 11134 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:35:10.617102 24960 raft_consensus.cc:482] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f2fdd596a8b44bbc848fe40f9f07fbe0)
I20240502 21:35:10.617380 24960 raft_consensus.cc:504] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:35:10.618643 24960 leader_election.cc:288] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467)
W20240502 21:35:10.620044 11297 leader_election.cc:334] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Network error: Client connection negotiation failed: client connection to 127.27.196.67:43203: connect: Connection refused (error 111)
W20240502 21:35:10.625006 11298 leader_election.cc:334] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111)
I20240502 21:35:10.625309 11298 leader_election.cc:302] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4cbb5b06a6a44eb6b2145cd538b01af2; no voters: 525a093ea5bf40dc9bbf9dc39895d670, f2fdd596a8b44bbc848fe40f9f07fbe0
I20240502 21:35:10.625679 24960 raft_consensus.cc:2726] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:35:10.631333 11493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ee481a3ece47a0905e93c66dbf949d" candidate_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "e0de0545fa0547d3929111e990e6d1b2" is_pre_election: true
I20240502 21:35:11.005424 24960 raft_consensus.cc:482] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 525a093ea5bf40dc9bbf9dc39895d670)
I20240502 21:35:11.005681 24960 raft_consensus.cc:504] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:35:11.006465 24960 leader_election.cc:288] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:35:11.008548 24974 raft_consensus.cc:482] T 58c7004ceb044f5fa9dec54910046b69 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 525a093ea5bf40dc9bbf9dc39895d670)
W20240502 21:35:11.008594 11298 leader_election.cc:334] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111)
W20240502 21:35:11.008729 11297 leader_election.cc:334] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Network error: Client connection negotiation failed: client connection to 127.27.196.67:43203: connect: Connection refused (error 111)
I20240502 21:35:11.008828 11297 leader_election.cc:302] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cbb5b06a6a44eb6b2145cd538b01af2; no voters: 525a093ea5bf40dc9bbf9dc39895d670, f2fdd596a8b44bbc848fe40f9f07fbe0
I20240502 21:35:11.008785 24974 raft_consensus.cc:504] T 58c7004ceb044f5fa9dec54910046b69 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:35:11.009128 24960 raft_consensus.cc:2726] T 8129d13163f64368b731ba4eaeaaa81c P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:35:11.009742 24974 leader_election.cc:288] T 58c7004ceb044f5fa9dec54910046b69 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931)
W20240502 21:35:11.011180 11432 leader_election.cc:334] T 58c7004ceb044f5fa9dec54910046b69 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111)
W20240502 21:35:11.011356 11429 leader_election.cc:334] T 58c7004ceb044f5fa9dec54910046b69 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): Network error: Client connection negotiation failed: client connection to 127.27.196.68:40931: connect: Connection refused (error 111)
I20240502 21:35:11.011467 11429 leader_election.cc:302] T 58c7004ceb044f5fa9dec54910046b69 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 19f162e164854f2da623a4bb8f9e6bf6, 525a093ea5bf40dc9bbf9dc39895d670
I20240502 21:35:11.011749 24974 raft_consensus.cc:2726] T 58c7004ceb044f5fa9dec54910046b69 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:35:11.027899 24976 raft_consensus.cc:482] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 525a093ea5bf40dc9bbf9dc39895d670)
I20240502 21:35:11.029347 24976 raft_consensus.cc:504] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:35:11.030251 24976 leader_election.cc:288] T 51936fb1f678458a8795f8246590addb P d10e9e156ba74a97a221c85b68ab8bc4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467)
I20240502 21:35:11.030601 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 11009
I20240502 21:35:11.056970 24993 raft_consensus.cc:482] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f2fdd596a8b44bbc848fe40f9f07fbe0)
I20240502 21:35:11.057287 24993 raft_consensus.cc:504] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:35:11.058303 24993 leader_election.cc:288] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
W20240502 21:35:11.059873 11297 leader_election.cc:334] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Network error: Client connection negotiation failed: client connection to 127.27.196.67:43203: connect: Connection refused (error 111)
W20240502 21:35:11.100962 11296 connection.cc:535] client connection to 127.27.196.69:42049 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:11.101176 11295 connection.cc:535] server connection from 127.27.196.69:42039 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:35:11.101531 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 11280
W20240502 21:35:11.101539 11429 connection.cc:535] server connection from 127.27.196.69:43471 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:11.102533 11296 leader_election.cc:334] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Network error: Client connection negotiation failed: client connection to 127.27.196.69:42049: connect: Connection refused (error 111)
I20240502 21:35:11.102712 11296 leader_election.cc:302] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cbb5b06a6a44eb6b2145cd538b01af2; no voters: d10e9e156ba74a97a221c85b68ab8bc4, f2fdd596a8b44bbc848fe40f9f07fbe0
I20240502 21:35:11.103055 24993 raft_consensus.cc:2726] T aeee59c5512d448d8f986f959dce1cc8 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:35:11.108706 11404 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:35:11.109686  8946 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111) [suppressed 106 similar messages]
W20240502 21:35:11.111377  8946 catalog_manager.cc:4573] TS 6ef0beaf2f6941afb6ea9f9f274517fe (127.27.196.70:44211): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet cdf72d25f0f847e5998acbc53979e64d: Network error: Client connection negotiation failed: client connection to 127.27.196.70:44211: connect: Connection refused (error 111)
W20240502 21:35:11.130290 11430 consensus_peers.cc:485] T a7c6b881c9ef4cdca787f065778e526f P e0de0545fa0547d3929111e990e6d1b2 -> Peer d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049): Couldn't send request to peer d10e9e156ba74a97a221c85b68ab8bc4. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.69:42049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:11.183755 24974 raft_consensus.cc:482] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 19f162e164854f2da623a4bb8f9e6bf6)
I20240502 21:35:11.184011 24974 raft_consensus.cc:504] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:35:11.184863 24974 leader_election.cc:288] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857)
W20240502 21:35:11.187721 11429 leader_election.cc:334] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): Network error: Client connection negotiation failed: client connection to 127.27.196.68:40931: connect: Connection refused (error 111)
W20240502 21:35:11.511669 11429 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Couldn't send request to peer 5543ddab5b484642b139b7c848932a58. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:35:11.514046 24974 raft_consensus.cc:482] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f2fdd596a8b44bbc848fe40f9f07fbe0)
I20240502 21:35:11.514271 24974 raft_consensus.cc:504] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:35:11.514649 24999 raft_consensus.cc:482] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f2fdd596a8b44bbc848fe40f9f07fbe0)
I20240502 21:35:11.514856 24999 raft_consensus.cc:504] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "5543ddab5b484642b139b7c848932a58" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42177 } }
I20240502 21:35:11.515312 24974 leader_election.cc:288] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203)
I20240502 21:35:11.515551 24974 raft_consensus.cc:482] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f2fdd596a8b44bbc848fe40f9f07fbe0)
I20240502 21:35:11.519084 24960 raft_consensus.cc:482] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f2fdd596a8b44bbc848fe40f9f07fbe0)
I20240502 21:35:11.519347 24960 raft_consensus.cc:504] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:35:11.520263 24960 leader_election.cc:288] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467)
W20240502 21:35:11.524487 11431 leader_election.cc:334] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Network error: Client connection negotiation failed: client connection to 127.27.196.67:43203: connect: Connection refused (error 111)
I20240502 21:35:11.525964 24999 leader_election.cc:288] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177)
I20240502 21:35:11.515759 24974 raft_consensus.cc:504] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
W20240502 21:35:11.526609 11431 leader_election.cc:334] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Network error: Client connection negotiation failed: client connection to 127.27.196.67:43203: connect: Connection refused (error 111)
I20240502 21:35:11.527364 24974 leader_election.cc:288] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857)
I20240502 21:35:11.528257 25001 raft_consensus.cc:482] T 1497b781f36c43b39d22a05bb524c59d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 19f162e164854f2da623a4bb8f9e6bf6)
W20240502 21:35:11.528399 11431 leader_election.cc:334] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Network error: Client connection negotiation failed: client connection to 127.27.196.67:43203: connect: Connection refused (error 111)
I20240502 21:35:11.528463 25001 raft_consensus.cc:504] T 1497b781f36c43b39d22a05bb524c59d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:35:11.529785 25001 leader_election.cc:288] T 1497b781f36c43b39d22a05bb524c59d P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), e0de0545fa0547d3929111e990e6d1b2 (127.27.196.72:40467)
W20240502 21:35:11.530066 11429 leader_election.cc:334] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5543ddab5b484642b139b7c848932a58 (127.27.196.65:42177): Network error: Client connection negotiation failed: client connection to 127.27.196.65:42177: connect: Connection refused (error 111)
I20240502 21:35:11.530263 11429 leader_election.cc:302] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 5543ddab5b484642b139b7c848932a58, f2fdd596a8b44bbc848fe40f9f07fbe0
W20240502 21:35:11.530995 11295 leader_election.cc:334] T 1497b781f36c43b39d22a05bb524c59d P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): Network error: Client connection negotiation failed: client connection to 127.27.196.68:40931: connect: Connection refused (error 111)
I20240502 21:35:11.531458 24974 raft_consensus.cc:482] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f2fdd596a8b44bbc848fe40f9f07fbe0)
I20240502 21:35:11.531788 25000 raft_consensus.cc:2726] T 4674e376bf7440cda619784491685b44 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:35:11.531651 24974 raft_consensus.cc:504] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:35:11.531858 11365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ee481a3ece47a0905e93c66dbf949d" candidate_uuid: "e0de0545fa0547d3929111e990e6d1b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" is_pre_election: true
I20240502 21:35:11.531811 11363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1497b781f36c43b39d22a05bb524c59d" candidate_uuid: "e0de0545fa0547d3929111e990e6d1b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" is_pre_election: true
I20240502 21:35:11.532277 11365 raft_consensus.cc:2454] T 41ee481a3ece47a0905e93c66dbf949d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0de0545fa0547d3929111e990e6d1b2 in term 1.
I20240502 21:35:11.532414 24974 leader_election.cc:288] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857)
I20240502 21:35:11.532656 11363 raft_consensus.cc:2454] T 1497b781f36c43b39d22a05bb524c59d P 4cbb5b06a6a44eb6b2145cd538b01af2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0de0545fa0547d3929111e990e6d1b2 in term 1.
W20240502 21:35:11.533126 11297 leader_election.cc:334] T 87f35aff2b104c01a9d9d48a7e9c9b23 P 4cbb5b06a6a44eb6b2145cd538b01af2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Network error: Client connection negotiation failed: client connection to 127.27.196.67:43203: connect: Connection refused (error 111)
I20240502 21:35:11.533771 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 11280
I20240502 21:35:11.535987 11493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f35aff2b104c01a9d9d48a7e9c9b23" candidate_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "e0de0545fa0547d3929111e990e6d1b2" is_pre_election: true
I20240502 21:35:11.536273 11493 raft_consensus.cc:2454] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cbb5b06a6a44eb6b2145cd538b01af2 in term 2.
I20240502 21:35:11.536326 11429 leader_election.cc:302] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cbb5b06a6a44eb6b2145cd538b01af2, e0de0545fa0547d3929111e990e6d1b2; no voters: 19f162e164854f2da623a4bb8f9e6bf6
I20240502 21:35:11.536854 24974 raft_consensus.cc:2781] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:35:11.536971 24974 raft_consensus.cc:482] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 19f162e164854f2da623a4bb8f9e6bf6)
I20240502 21:35:11.537084 24974 raft_consensus.cc:3037] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:35:11.537089 11432 leader_election.cc:334] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111)
I20240502 21:35:11.537621 11432 leader_election.cc:302] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 525a093ea5bf40dc9bbf9dc39895d670, f2fdd596a8b44bbc848fe40f9f07fbe0
W20240502 21:35:11.537750 11431 leader_election.cc:334] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203): Network error: Client connection negotiation failed: client connection to 127.27.196.67:43203: connect: Connection refused (error 111)
I20240502 21:35:11.538005 25000 raft_consensus.cc:2726] T cffcb4f7f7754262beeb06168097f6dc P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:11.538229 11432 leader_election.cc:334] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111)
I20240502 21:35:11.538398 11432 leader_election.cc:302] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4cbb5b06a6a44eb6b2145cd538b01af2, e0de0545fa0547d3929111e990e6d1b2; no voters: 525a093ea5bf40dc9bbf9dc39895d670, f2fdd596a8b44bbc848fe40f9f07fbe0
I20240502 21:35:11.538893 25000 raft_consensus.cc:2726] T 41ee481a3ece47a0905e93c66dbf949d P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:35:11.540418 24974 raft_consensus.cc:504] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:35:11.541093 24974 leader_election.cc:288] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 election: Requested vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857)
W20240502 21:35:11.542403 11429 leader_election.cc:334] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931): Network error: Client connection negotiation failed: client connection to 127.27.196.68:40931: connect: Connection refused (error 111)
W20240502 21:35:11.608669 11429 connection.cc:535] client connection to 127.27.196.71:35857 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:35:11.609081 11431 connection.cc:535] server connection from 127.27.196.71:57407 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:35:11.609077 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 11414
W20240502 21:35:11.610947 11429 leader_election.cc:334] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857): Network error: Client connection negotiation failed: client connection to 127.27.196.71:35857: connect: Connection refused (error 111)
I20240502 21:35:11.611183 11429 leader_election.cc:302] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 4cbb5b06a6a44eb6b2145cd538b01af2, f2fdd596a8b44bbc848fe40f9f07fbe0
W20240502 21:35:11.611696 11429 leader_election.cc:334] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857): Network error: Client connection negotiation failed: client connection to 127.27.196.71:35857: connect: Connection refused (error 111)
I20240502 21:35:11.611896 24974 raft_consensus.cc:2726] T 87f35aff2b104c01a9d9d48a7e9c9b23 P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 21:35:11.611864 11429 leader_election.cc:302] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0de0545fa0547d3929111e990e6d1b2; no voters: 19f162e164854f2da623a4bb8f9e6bf6, 4cbb5b06a6a44eb6b2145cd538b01af2
I20240502 21:35:11.612249 25000 raft_consensus.cc:2726] T 1497b781f36c43b39d22a05bb524c59d P e0de0545fa0547d3929111e990e6d1b2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 21:35:11.616735 11536 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:35:12.019749 24999 raft_consensus.cc:482] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 525a093ea5bf40dc9bbf9dc39895d670)
I20240502 21:35:12.020713 25000 raft_consensus.cc:482] T a2f3edaf1c874cefb820f027f90d1185 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 19f162e164854f2da623a4bb8f9e6bf6)
I20240502 21:35:12.020906 25000 raft_consensus.cc:504] T a2f3edaf1c874cefb820f027f90d1185 P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "19f162e164854f2da623a4bb8f9e6bf6" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 40931 } } peers { permanent_uuid: "f2fdd596a8b44bbc848fe40f9f07fbe0" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 43203 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } } peers { permanent_uuid: "4cbb5b06a6a44eb6b2145cd538b01af2" member_type: VOTER last_known_addr { host: "127.27.196.71" port: 35857 } }
I20240502 21:35:12.022954 25000 leader_election.cc:288] T a2f3edaf1c874cefb820f027f90d1185 P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19f162e164854f2da623a4bb8f9e6bf6 (127.27.196.68:40931), f2fdd596a8b44bbc848fe40f9f07fbe0 (127.27.196.67:43203), 4cbb5b06a6a44eb6b2145cd538b01af2 (127.27.196.71:35857)
W20240502 21:35:12.022948 11432 consensus_peers.cc:485] T 94d2cdf8ae6941668b5f680c558656e8 P e0de0545fa0547d3929111e990e6d1b2 -> Peer 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071): Couldn't send request to peer 525a093ea5bf40dc9bbf9dc39895d670. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.66:39071: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:35:12.024171 24999 raft_consensus.cc:504] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "525a093ea5bf40dc9bbf9dc39895d670" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 39071 } } peers { permanent_uuid: "d10e9e156ba74a97a221c85b68ab8bc4" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 42049 } } peers { permanent_uuid: "e0de0545fa0547d3929111e990e6d1b2" member_type: VOTER last_known_addr { host: "127.27.196.72" port: 40467 } }
I20240502 21:35:12.025067 24999 leader_election.cc:288] T 51936fb1f678458a8795f8246590addb P e0de0545fa0547d3929111e990e6d1b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 525a093ea5bf40dc9bbf9dc39895d670 (127.27.196.66:39071), d10e9e156ba74a97a221c85b68ab8bc4 (127.27.196.69:42049)
I20240502 21:35:12.027312 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 11414
I20240502 21:35:12.096514 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 8930
I20240502 21:35:12.103364  8991 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:35:12.490141 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 8930
2024-05-02T21:35:12Z chronyd exiting
I20240502 21:35:12.576421 28433 test_util.cc:175] -----------------------------------------------
I20240502 21:35:12.576527 28433 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714685119845993-28433-0
[  FAILED  ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0, where GetParam() = 4-byte object <00-00 00-00> (392328 ms)
[----------] 1 test from ConcurrentRebalancersTest (392328 ms total)

[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN      ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/0
I20240502 21:35:12.579172 28433 test_util.cc:268] Using random seed: 1624029723
I20240502 21:35:12.581429 28433 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:35:12.581508 28433 ts_itest-base.cc:113] --------------
I20240502 21:35:12.581550 28433 ts_itest-base.cc:114] 4 tablet servers
I20240502 21:35:12.581593 28433 ts_itest-base.cc:115] 1 replicas per TS
I20240502 21:35:12.581648 28433 ts_itest-base.cc:116] --------------
2024-05-02T21:35:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:35:12Z Disabled control of system clock
I20240502 21:35:12.660977 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.196.126:43485
--webserver_interface=127.27.196.126
--webserver_port=0
--builtin_ntp_servers=127.27.196.84:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.196.126:43485
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240502 21:35:13.079236 25021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:35:13.079609 25021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:35:13.079725 25021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:35:13.087822 25021 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:35:13.087994 25021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:35:13.088057 25021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:35:13.088117 25021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:35:13.088165 25021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:35:13.098110 25021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.27.196.126:43485
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.196.126:43485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.27.196.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:35:13.099602 25021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:35:13.101517 25021 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:35:13.114116 25029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:13.114751 25027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:13.115052 25026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:35:13.115633 25021 server_base.cc:1034] running on GCE node
I20240502 21:35:13.117139 25021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:35:13.119058 25021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:35:13.120414 25021 hybrid_clock.cc:648] HybridClock initialized: now 1714685713120364 us; error 47 us; skew 500 ppm
I20240502 21:35:13.120898 25021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:35:13.123812 25021 webserver.cc:460] Webserver started at http://127.27.196.126:45399/ using document root <none> and password file <none>
I20240502 21:35:13.124727 25021 fs_manager.cc:359] Metadata directory not provided
I20240502 21:35:13.124872 25021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:35:13.125236 25021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:35:13.128448 25021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "cdc7fe5ec4d94cb4b14b96d243b6362c"
format_stamp: "Formatted at 2024-05-02 21:35:13 on dist-test-slave-3bs1"
I20240502 21:35:13.129381 25021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "cdc7fe5ec4d94cb4b14b96d243b6362c"
format_stamp: "Formatted at 2024-05-02 21:35:13 on dist-test-slave-3bs1"
I20240502 21:35:13.136292 25021 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.007s	sys 0.000s
I20240502 21:35:13.141162 25035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:13.143005 25021 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.005s	sys 0.000s
I20240502 21:35:13.143494 25021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
uuid: "cdc7fe5ec4d94cb4b14b96d243b6362c"
format_stamp: "Formatted at 2024-05-02 21:35:13 on dist-test-slave-3bs1"
I20240502 21:35:13.143908 25021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:35:13.165828 25021 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:35:13.166844 25021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:35:13.167229 25021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:35:13.189417 25021 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.126:43485
I20240502 21:35:13.189476 25087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.126:43485 every 8 connection(s)
I20240502 21:35:13.191031 25021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 21:35:13.196223 25088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:13.198767 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 25021
I20240502 21:35:13.199080 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:35:13.209172 25088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc7fe5ec4d94cb4b14b96d243b6362c: Bootstrap starting.
I20240502 21:35:13.212884 25088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdc7fe5ec4d94cb4b14b96d243b6362c: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:13.214537 25088 log.cc:824] T 00000000000000000000000000000000 P cdc7fe5ec4d94cb4b14b96d243b6362c: Log is configured to *not* fsync() on all Append() calls
I20240502 21:35:13.217856 25088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc7fe5ec4d94cb4b14b96d243b6362c: No bootstrap required, opened a new log
I20240502 21:35:13.225379 25088 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdc7fe5ec4d94cb4b14b96d243b6362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7fe5ec4d94cb4b14b96d243b6362c" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 43485 } }
I20240502 21:35:13.226140 25088 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdc7fe5ec4d94cb4b14b96d243b6362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:13.226330 25088 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdc7fe5ec4d94cb4b14b96d243b6362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc7fe5ec4d94cb4b14b96d243b6362c, State: Initialized, Role: FOLLOWER
I20240502 21:35:13.227164 25088 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdc7fe5ec4d94cb4b14b96d243b6362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7fe5ec4d94cb4b14b96d243b6362c" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 43485 } }
I20240502 21:35:13.227429 25088 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdc7fe5ec4d94cb4b14b96d243b6362c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:35:13.227584 25088 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdc7fe5ec4d94cb4b14b96d243b6362c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:35:13.227929 25088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdc7fe5ec4d94cb4b14b96d243b6362c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:13.230729 25088 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdc7fe5ec4d94cb4b14b96d243b6362c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7fe5ec4d94cb4b14b96d243b6362c" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 43485 } }
I20240502 21:35:13.231408 25088 leader_election.cc:302] T 00000000000000000000000000000000 P cdc7fe5ec4d94cb4b14b96d243b6362c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdc7fe5ec4d94cb4b14b96d243b6362c; no voters: 
I20240502 21:35:13.232319 25088 leader_election.cc:288] T 00000000000000000000000000000000 P cdc7fe5ec4d94cb4b14b96d243b6362c [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 21:35:13.232595 25093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdc7fe5ec4d94cb4b14b96d243b6362c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:13.233486 25093 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdc7fe5ec4d94cb4b14b96d243b6362c [term 1 LEADER]: Becoming Leader. State: Replica: cdc7fe5ec4d94cb4b14b96d243b6362c, State: Running, Role: LEADER
I20240502 21:35:13.234097 25093 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdc7fe5ec4d94cb4b14b96d243b6362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7fe5ec4d94cb4b14b96d243b6362c" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 43485 } }
I20240502 21:35:13.234820 25088 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdc7fe5ec4d94cb4b14b96d243b6362c [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:35:13.238452 25094 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc7fe5ec4d94cb4b14b96d243b6362c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdc7fe5ec4d94cb4b14b96d243b6362c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7fe5ec4d94cb4b14b96d243b6362c" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 43485 } } }
I20240502 21:35:13.238922 25094 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc7fe5ec4d94cb4b14b96d243b6362c [sys.catalog]: This master's current role is: LEADER
I20240502 21:35:13.239568 25095 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc7fe5ec4d94cb4b14b96d243b6362c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdc7fe5ec4d94cb4b14b96d243b6362c. Latest consensus state: current_term: 1 leader_uuid: "cdc7fe5ec4d94cb4b14b96d243b6362c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7fe5ec4d94cb4b14b96d243b6362c" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 43485 } } }
I20240502 21:35:13.240020 25095 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc7fe5ec4d94cb4b14b96d243b6362c [sys.catalog]: This master's current role is: LEADER
I20240502 21:35:13.241946 25103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:35:13.248937 25103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:35:13.261245 25103 catalog_manager.cc:1347] Generated new cluster ID: 44b9c60aa26640e2ae47d5217655d3f5
I20240502 21:35:13.261420 25103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:35:13.276854 25103 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:35:13.278579 25103 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:35:13.291508 25103 catalog_manager.cc:5857] T 00000000000000000000000000000000 P cdc7fe5ec4d94cb4b14b96d243b6362c: Generated new TSK 0
I20240502 21:35:13.292501 25103 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:35:13.303237 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.65:0
--local_ip_for_outbound_sockets=127.27.196.65
--webserver_interface=127.27.196.65
--webserver_port=0
--tserver_master_addrs=127.27.196.126:43485
--builtin_ntp_servers=127.27.196.84:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 21:35:13.735922 25112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:35:13.736377 25112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:35:13.736549 25112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:35:13.746670 25112 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:35:13.746871 25112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:35:13.747081 25112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.65
I20240502 21:35:13.758030 25112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.27.196.65
--webserver_port=0
--tserver_master_addrs=127.27.196.126:43485
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.65
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:35:13.759615 25112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:35:13.761598 25112 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:35:13.776686 25117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:13.777577 25118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:35:13.778479 25112 server_base.cc:1034] running on GCE node
W20240502 21:35:13.778669 25120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:35:13.779682 25112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:35:13.780840 25112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:35:13.782074 25112 hybrid_clock.cc:648] HybridClock initialized: now 1714685713782012 us; error 70 us; skew 500 ppm
I20240502 21:35:13.782454 25112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:35:13.785539 25112 webserver.cc:460] Webserver started at http://127.27.196.65:35565/ using document root <none> and password file <none>
I20240502 21:35:13.786434 25112 fs_manager.cc:359] Metadata directory not provided
I20240502 21:35:13.786562 25112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:35:13.786932 25112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:35:13.790176 25112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc"
format_stamp: "Formatted at 2024-05-02 21:35:13 on dist-test-slave-3bs1"
I20240502 21:35:13.791209 25112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc"
format_stamp: "Formatted at 2024-05-02 21:35:13 on dist-test-slave-3bs1"
I20240502 21:35:13.798601 25112 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.005s	sys 0.000s
I20240502 21:35:13.804078 25126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:13.806260 25112 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
I20240502 21:35:13.806591 25112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc"
format_stamp: "Formatted at 2024-05-02 21:35:13 on dist-test-slave-3bs1"
I20240502 21:35:13.806926 25112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:35:13.845882 25112 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:35:13.847011 25112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:35:13.847409 25112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:35:13.848763 25112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:35:13.850811 25112 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:35:13.850930 25112 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:13.851068 25112 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:35:13.851172 25112 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.001s	sys 0.000s
I20240502 21:35:13.892087 25112 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.65:32813
I20240502 21:35:13.892165 25241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.65:32813 every 8 connection(s)
I20240502 21:35:13.893635 25112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:35:13.902865 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 25112
I20240502 21:35:13.903131 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:35:13.907955 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.66:0
--local_ip_for_outbound_sockets=127.27.196.66
--webserver_interface=127.27.196.66
--webserver_port=0
--tserver_master_addrs=127.27.196.126:43485
--builtin_ntp_servers=127.27.196.84:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:35:13.908742 25242 heartbeater.cc:347] Connected to a master server at 127.27.196.126:43485
I20240502 21:35:13.909251 25242 heartbeater.cc:464] Registering TS with master...
I20240502 21:35:13.910394 25242 heartbeater.cc:511] Master 127.27.196.126:43485 requested a full tablet report, sending...
I20240502 21:35:13.913657 25052 ts_manager.cc:194] Registered new tserver with Master: f638cb4a3c7f4d368fff500a0cb1e3bc (127.27.196.65:32813)
I20240502 21:35:13.916110 25052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.65:58945
W20240502 21:35:14.336622 25246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:35:14.336982 25246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:35:14.337112 25246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:35:14.345526 25246 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:35:14.345687 25246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:35:14.345865 25246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.66
I20240502 21:35:14.355878 25246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.27.196.66
--webserver_port=0
--tserver_master_addrs=127.27.196.126:43485
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.66
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:35:14.357354 25246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:35:14.359263 25246 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:35:14.373865 25251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:14.373946 25254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:14.374459 25252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:35:14.374975 25246 server_base.cc:1034] running on GCE node
I20240502 21:35:14.375694 25246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:35:14.376780 25246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:35:14.378016 25246 hybrid_clock.cc:648] HybridClock initialized: now 1714685714377956 us; error 55 us; skew 500 ppm
I20240502 21:35:14.378425 25246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:35:14.382220 25246 webserver.cc:460] Webserver started at http://127.27.196.66:44161/ using document root <none> and password file <none>
I20240502 21:35:14.383384 25246 fs_manager.cc:359] Metadata directory not provided
I20240502 21:35:14.383544 25246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:35:14.383953 25246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:35:14.390862 25246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "c09724fc20c4461981288a0c25a03743"
format_stamp: "Formatted at 2024-05-02 21:35:14 on dist-test-slave-3bs1"
I20240502 21:35:14.391935 25246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "c09724fc20c4461981288a0c25a03743"
format_stamp: "Formatted at 2024-05-02 21:35:14 on dist-test-slave-3bs1"
I20240502 21:35:14.398424 25246 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.000s
I20240502 21:35:14.402805 25260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:14.404873 25246 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 21:35:14.405078 25246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c09724fc20c4461981288a0c25a03743"
format_stamp: "Formatted at 2024-05-02 21:35:14 on dist-test-slave-3bs1"
I20240502 21:35:14.405328 25246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:35:14.430881 25246 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:35:14.431948 25246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:35:14.432253 25246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:35:14.433569 25246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:35:14.435415 25246 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:35:14.435531 25246 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:14.435654 25246 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:35:14.435731 25246 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:14.475639 25246 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.66:45995
I20240502 21:35:14.475665 25374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.66:45995 every 8 connection(s)
I20240502 21:35:14.477113 25246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:35:14.479975 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 25246
I20240502 21:35:14.480176 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:35:14.484373 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.67:0
--local_ip_for_outbound_sockets=127.27.196.67
--webserver_interface=127.27.196.67
--webserver_port=0
--tserver_master_addrs=127.27.196.126:43485
--builtin_ntp_servers=127.27.196.84:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:35:14.498235 25375 heartbeater.cc:347] Connected to a master server at 127.27.196.126:43485
I20240502 21:35:14.498643 25375 heartbeater.cc:464] Registering TS with master...
I20240502 21:35:14.499624 25375 heartbeater.cc:511] Master 127.27.196.126:43485 requested a full tablet report, sending...
I20240502 21:35:14.501297 25052 ts_manager.cc:194] Registered new tserver with Master: c09724fc20c4461981288a0c25a03743 (127.27.196.66:45995)
I20240502 21:35:14.502336 25052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.66:35537
W20240502 21:35:14.911561 25379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:35:14.911909 25379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:35:14.912060 25379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:35:14.920568 25379 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
I20240502 21:35:14.920503 25242 heartbeater.cc:503] Master 127.27.196.126:43485 was elected leader, sending a full tablet report...
W20240502 21:35:14.920728 25379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:35:14.920928 25379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.67
I20240502 21:35:14.931397 25379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.27.196.67
--webserver_port=0
--tserver_master_addrs=127.27.196.126:43485
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.67
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:35:14.932972 25379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:35:14.934934 25379 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:35:14.949013 25384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:14.948989 25385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:14.949633 25387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:35:14.950109 25379 server_base.cc:1034] running on GCE node
I20240502 21:35:14.950913 25379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:35:14.952172 25379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:35:14.953575 25379 hybrid_clock.cc:648] HybridClock initialized: now 1714685714953512 us; error 110 us; skew 500 ppm
I20240502 21:35:14.953991 25379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:35:14.956902 25379 webserver.cc:460] Webserver started at http://127.27.196.67:41135/ using document root <none> and password file <none>
I20240502 21:35:14.957755 25379 fs_manager.cc:359] Metadata directory not provided
I20240502 21:35:14.957877 25379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:35:14.958227 25379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:35:14.961145 25379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9190116ee076433daec7b155ae8e5e05"
format_stamp: "Formatted at 2024-05-02 21:35:14 on dist-test-slave-3bs1"
I20240502 21:35:14.962396 25379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9190116ee076433daec7b155ae8e5e05"
format_stamp: "Formatted at 2024-05-02 21:35:14 on dist-test-slave-3bs1"
I20240502 21:35:14.969343 25379 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.009s	sys 0.000s
I20240502 21:35:14.973726 25393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:14.976215 25379 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.001s
I20240502 21:35:14.976492 25379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9190116ee076433daec7b155ae8e5e05"
format_stamp: "Formatted at 2024-05-02 21:35:14 on dist-test-slave-3bs1"
I20240502 21:35:14.976819 25379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:35:15.029157 25379 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:35:15.030210 25379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:35:15.030539 25379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:35:15.031941 25379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:35:15.033937 25379 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:35:15.034049 25379 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:15.034174 25379 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:35:15.034246 25379 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:15.072933 25379 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.67:33249
I20240502 21:35:15.073035 25508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.67:33249 every 8 connection(s)
I20240502 21:35:15.074440 25379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:35:15.080159 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 25379
I20240502 21:35:15.080385 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:35:15.084199 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.68:0
--local_ip_for_outbound_sockets=127.27.196.68
--webserver_interface=127.27.196.68
--webserver_port=0
--tserver_master_addrs=127.27.196.126:43485
--builtin_ntp_servers=127.27.196.84:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:35:15.089346 25509 heartbeater.cc:347] Connected to a master server at 127.27.196.126:43485
I20240502 21:35:15.089762 25509 heartbeater.cc:464] Registering TS with master...
I20240502 21:35:15.090829 25509 heartbeater.cc:511] Master 127.27.196.126:43485 requested a full tablet report, sending...
I20240502 21:35:15.092728 25052 ts_manager.cc:194] Registered new tserver with Master: 9190116ee076433daec7b155ae8e5e05 (127.27.196.67:33249)
I20240502 21:35:15.093601 25052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.67:38999
W20240502 21:35:15.476521 25513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:35:15.476822 25513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:35:15.476958 25513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:35:15.485426 25513 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:35:15.485606 25513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:35:15.485780 25513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.68
I20240502 21:35:15.495818 25513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.27.196.68
--webserver_port=0
--tserver_master_addrs=127.27.196.126:43485
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.68
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:35:15.497361 25513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:35:15.499601 25513 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 21:35:15.506048 25375 heartbeater.cc:503] Master 127.27.196.126:43485 was elected leader, sending a full tablet report...
W20240502 21:35:15.514494 25518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:15.515323 25521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:15.515966 25519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:35:15.516438 25513 server_base.cc:1034] running on GCE node
I20240502 21:35:15.517203 25513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:35:15.519022 25513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:35:15.520350 25513 hybrid_clock.cc:648] HybridClock initialized: now 1714685715520284 us; error 65 us; skew 500 ppm
I20240502 21:35:15.520864 25513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:35:15.523855 25513 webserver.cc:460] Webserver started at http://127.27.196.68:33919/ using document root <none> and password file <none>
I20240502 21:35:15.524685 25513 fs_manager.cc:359] Metadata directory not provided
I20240502 21:35:15.524806 25513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:35:15.525089 25513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:35:15.528008 25513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "0b98de82262a426980c54b3b294dbcb5"
format_stamp: "Formatted at 2024-05-02 21:35:15 on dist-test-slave-3bs1"
I20240502 21:35:15.528837 25513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "0b98de82262a426980c54b3b294dbcb5"
format_stamp: "Formatted at 2024-05-02 21:35:15 on dist-test-slave-3bs1"
I20240502 21:35:15.534658 25513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.001s
I20240502 21:35:15.538559 25527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:15.540184 25513 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 21:35:15.540395 25513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "0b98de82262a426980c54b3b294dbcb5"
format_stamp: "Formatted at 2024-05-02 21:35:15 on dist-test-slave-3bs1"
I20240502 21:35:15.540633 25513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:35:15.577428 25513 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:35:15.578364 25513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:35:15.578653 25513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:35:15.579747 25513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:35:15.581494 25513 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:35:15.581597 25513 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:15.581717 25513 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:35:15.581789 25513 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:15.618410 25513 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.68:35175
I20240502 21:35:15.618502 25643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.68:35175 every 8 connection(s)
I20240502 21:35:15.619912 25513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:35:15.621490 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 25513
I20240502 21:35:15.621706 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 21:35:15.640297 25644 heartbeater.cc:347] Connected to a master server at 127.27.196.126:43485
I20240502 21:35:15.640709 25644 heartbeater.cc:464] Registering TS with master...
I20240502 21:35:15.641649 25644 heartbeater.cc:511] Master 127.27.196.126:43485 requested a full tablet report, sending...
I20240502 21:35:15.643464 25051 ts_manager.cc:194] Registered new tserver with Master: 0b98de82262a426980c54b3b294dbcb5 (127.27.196.68:35175)
I20240502 21:35:15.644515 25051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.68:52917
I20240502 21:35:15.653954 28433 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 21:35:15.675722 25051 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51332:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240502 21:35:15.723321 25174 tablet_service.cc:1467] Processing CreateTablet for tablet f2229e07dab648ba8126efd4e470ae61 (DEFAULT_TABLE table=TestTable [id=7a179d866cae4bd4b70eb8f540b95927]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:15.725359 25174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2229e07dab648ba8126efd4e470ae61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:15.743847 25659 tablet_bootstrap.cc:492] T f2229e07dab648ba8126efd4e470ae61 P f638cb4a3c7f4d368fff500a0cb1e3bc: Bootstrap starting.
I20240502 21:35:15.747696 25659 tablet_bootstrap.cc:654] T f2229e07dab648ba8126efd4e470ae61 P f638cb4a3c7f4d368fff500a0cb1e3bc: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:15.748852 25659 log.cc:824] T f2229e07dab648ba8126efd4e470ae61 P f638cb4a3c7f4d368fff500a0cb1e3bc: Log is configured to *not* fsync() on all Append() calls
I20240502 21:35:15.752349 25659 tablet_bootstrap.cc:492] T f2229e07dab648ba8126efd4e470ae61 P f638cb4a3c7f4d368fff500a0cb1e3bc: No bootstrap required, opened a new log
I20240502 21:35:15.752652 25659 ts_tablet_manager.cc:1367] T f2229e07dab648ba8126efd4e470ae61 P f638cb4a3c7f4d368fff500a0cb1e3bc: Time spent bootstrapping tablet: real 0.009s	user 0.006s	sys 0.001s
I20240502 21:35:15.759423 25659 raft_consensus.cc:348] T f2229e07dab648ba8126efd4e470ae61 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } }
I20240502 21:35:15.759913 25659 raft_consensus.cc:374] T f2229e07dab648ba8126efd4e470ae61 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:15.760056 25659 raft_consensus.cc:729] T f2229e07dab648ba8126efd4e470ae61 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f638cb4a3c7f4d368fff500a0cb1e3bc, State: Initialized, Role: FOLLOWER
I20240502 21:35:15.760834 25659 consensus_queue.cc:260] T f2229e07dab648ba8126efd4e470ae61 P f638cb4a3c7f4d368fff500a0cb1e3bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } }
I20240502 21:35:15.761104 25659 raft_consensus.cc:388] T f2229e07dab648ba8126efd4e470ae61 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:35:15.761255 25659 raft_consensus.cc:482] T f2229e07dab648ba8126efd4e470ae61 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:35:15.761428 25659 raft_consensus.cc:3037] T f2229e07dab648ba8126efd4e470ae61 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:15.764304 25659 raft_consensus.cc:504] T f2229e07dab648ba8126efd4e470ae61 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } }
I20240502 21:35:15.764894 25659 leader_election.cc:302] T f2229e07dab648ba8126efd4e470ae61 P f638cb4a3c7f4d368fff500a0cb1e3bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f638cb4a3c7f4d368fff500a0cb1e3bc; no voters: 
I20240502 21:35:15.765991 25659 leader_election.cc:288] T f2229e07dab648ba8126efd4e470ae61 P f638cb4a3c7f4d368fff500a0cb1e3bc [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 21:35:15.766273 25661 raft_consensus.cc:2781] T f2229e07dab648ba8126efd4e470ae61 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:15.767843 25661 raft_consensus.cc:686] T f2229e07dab648ba8126efd4e470ae61 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 1 LEADER]: Becoming Leader. State: Replica: f638cb4a3c7f4d368fff500a0cb1e3bc, State: Running, Role: LEADER
I20240502 21:35:15.768484 25661 consensus_queue.cc:237] T f2229e07dab648ba8126efd4e470ae61 P f638cb4a3c7f4d368fff500a0cb1e3bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } }
I20240502 21:35:15.769083 25659 ts_tablet_manager.cc:1398] T f2229e07dab648ba8126efd4e470ae61 P f638cb4a3c7f4d368fff500a0cb1e3bc: Time spent starting tablet: real 0.016s	user 0.003s	sys 0.013s
I20240502 21:35:15.775712 25051 catalog_manager.cc:5484] T f2229e07dab648ba8126efd4e470ae61 P f638cb4a3c7f4d368fff500a0cb1e3bc reported cstate change: term changed from 0 to 1, leader changed from <none> to f638cb4a3c7f4d368fff500a0cb1e3bc (127.27.196.65). New cstate: current_term: 1 leader_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } } }
I20240502 21:35:15.803509 28433 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 21:35:15.806591 28433 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f638cb4a3c7f4d368fff500a0cb1e3bc to finish bootstrapping
I20240502 21:35:15.817492 28433 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c09724fc20c4461981288a0c25a03743 to finish bootstrapping
I20240502 21:35:15.829154 28433 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 9190116ee076433daec7b155ae8e5e05 to finish bootstrapping
I20240502 21:35:15.840713 28433 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 0b98de82262a426980c54b3b294dbcb5 to finish bootstrapping
I20240502 21:35:15.853438 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 25379
I20240502 21:35:15.854995 25504 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:35:15.948279 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 25379
I20240502 21:35:15.969516 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 25513
I20240502 21:35:15.970808 25639 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:35:16.056983 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 25513
I20240502 21:35:19.830242 25051 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51332:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 21:35:19.842411 25174 tablet_service.cc:1467] Processing CreateTablet for tablet f37a95aae77446a7a765afc57f5b5732 (DEFAULT_TABLE table=rebalance_test_table_0 [id=954d6bb2eec945c09279d4a8b8c42193]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:19.842895 25174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f37a95aae77446a7a765afc57f5b5732. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:19.849507 25678 tablet_bootstrap.cc:492] T f37a95aae77446a7a765afc57f5b5732 P f638cb4a3c7f4d368fff500a0cb1e3bc: Bootstrap starting.
I20240502 21:35:19.851636 25678 tablet_bootstrap.cc:654] T f37a95aae77446a7a765afc57f5b5732 P f638cb4a3c7f4d368fff500a0cb1e3bc: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:19.853904 25678 tablet_bootstrap.cc:492] T f37a95aae77446a7a765afc57f5b5732 P f638cb4a3c7f4d368fff500a0cb1e3bc: No bootstrap required, opened a new log
I20240502 21:35:19.854084 25678 ts_tablet_manager.cc:1367] T f37a95aae77446a7a765afc57f5b5732 P f638cb4a3c7f4d368fff500a0cb1e3bc: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 21:35:19.853657 25307 tablet_service.cc:1467] Processing CreateTablet for tablet 6b25593b6b044eebb796d73d877199e4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=954d6bb2eec945c09279d4a8b8c42193]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:19.853690 25308 tablet_service.cc:1467] Processing CreateTablet for tablet 10c2741fd1924b69a8f98ef874a85610 (DEFAULT_TABLE table=rebalance_test_table_0 [id=954d6bb2eec945c09279d4a8b8c42193]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:19.855223 25307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b25593b6b044eebb796d73d877199e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:19.855171 25678 raft_consensus.cc:348] T f37a95aae77446a7a765afc57f5b5732 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } }
I20240502 21:35:19.855379 25678 raft_consensus.cc:374] T f37a95aae77446a7a765afc57f5b5732 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:19.855475 25678 raft_consensus.cc:729] T f37a95aae77446a7a765afc57f5b5732 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f638cb4a3c7f4d368fff500a0cb1e3bc, State: Initialized, Role: FOLLOWER
I20240502 21:35:19.855782 25308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10c2741fd1924b69a8f98ef874a85610. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:19.855780 25678 consensus_queue.cc:260] T f37a95aae77446a7a765afc57f5b5732 P f638cb4a3c7f4d368fff500a0cb1e3bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } }
I20240502 21:35:19.856079 25678 raft_consensus.cc:388] T f37a95aae77446a7a765afc57f5b5732 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:35:19.856222 25678 raft_consensus.cc:482] T f37a95aae77446a7a765afc57f5b5732 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:35:19.856389 25678 raft_consensus.cc:3037] T f37a95aae77446a7a765afc57f5b5732 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:19.858779 25678 raft_consensus.cc:504] T f37a95aae77446a7a765afc57f5b5732 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } }
I20240502 21:35:19.859042 25678 leader_election.cc:302] T f37a95aae77446a7a765afc57f5b5732 P f638cb4a3c7f4d368fff500a0cb1e3bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f638cb4a3c7f4d368fff500a0cb1e3bc; no voters: 
I20240502 21:35:19.859632 25678 leader_election.cc:288] T f37a95aae77446a7a765afc57f5b5732 P f638cb4a3c7f4d368fff500a0cb1e3bc [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 21:35:19.859776 25680 raft_consensus.cc:2781] T f37a95aae77446a7a765afc57f5b5732 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:19.866396 25680 raft_consensus.cc:686] T f37a95aae77446a7a765afc57f5b5732 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 1 LEADER]: Becoming Leader. State: Replica: f638cb4a3c7f4d368fff500a0cb1e3bc, State: Running, Role: LEADER
I20240502 21:35:19.866976 25678 ts_tablet_manager.cc:1398] T f37a95aae77446a7a765afc57f5b5732 P f638cb4a3c7f4d368fff500a0cb1e3bc: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.008s
I20240502 21:35:19.867297 25680 consensus_queue.cc:237] T f37a95aae77446a7a765afc57f5b5732 P f638cb4a3c7f4d368fff500a0cb1e3bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } }
I20240502 21:35:19.870249 25051 catalog_manager.cc:5484] T f37a95aae77446a7a765afc57f5b5732 P f638cb4a3c7f4d368fff500a0cb1e3bc reported cstate change: term changed from 0 to 1, leader changed from <none> to f638cb4a3c7f4d368fff500a0cb1e3bc (127.27.196.65). New cstate: current_term: 1 leader_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } } }
I20240502 21:35:19.871104 25685 tablet_bootstrap.cc:492] T 6b25593b6b044eebb796d73d877199e4 P c09724fc20c4461981288a0c25a03743: Bootstrap starting.
I20240502 21:35:19.877457 25685 tablet_bootstrap.cc:654] T 6b25593b6b044eebb796d73d877199e4 P c09724fc20c4461981288a0c25a03743: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:19.878803 25685 log.cc:824] T 6b25593b6b044eebb796d73d877199e4 P c09724fc20c4461981288a0c25a03743: Log is configured to *not* fsync() on all Append() calls
I20240502 21:35:19.881688 25685 tablet_bootstrap.cc:492] T 6b25593b6b044eebb796d73d877199e4 P c09724fc20c4461981288a0c25a03743: No bootstrap required, opened a new log
I20240502 21:35:19.881987 25685 ts_tablet_manager.cc:1367] T 6b25593b6b044eebb796d73d877199e4 P c09724fc20c4461981288a0c25a03743: Time spent bootstrapping tablet: real 0.011s	user 0.007s	sys 0.000s
I20240502 21:35:19.889082 25685 raft_consensus.cc:348] T 6b25593b6b044eebb796d73d877199e4 P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } }
I20240502 21:35:19.889470 25685 raft_consensus.cc:374] T 6b25593b6b044eebb796d73d877199e4 P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:19.889551 25685 raft_consensus.cc:729] T 6b25593b6b044eebb796d73d877199e4 P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c09724fc20c4461981288a0c25a03743, State: Initialized, Role: FOLLOWER
I20240502 21:35:19.890149 25685 consensus_queue.cc:260] T 6b25593b6b044eebb796d73d877199e4 P c09724fc20c4461981288a0c25a03743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } }
I20240502 21:35:19.890349 25685 raft_consensus.cc:388] T 6b25593b6b044eebb796d73d877199e4 P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:35:19.890463 25685 raft_consensus.cc:482] T 6b25593b6b044eebb796d73d877199e4 P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:35:19.890601 25685 raft_consensus.cc:3037] T 6b25593b6b044eebb796d73d877199e4 P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:19.892793 25685 raft_consensus.cc:504] T 6b25593b6b044eebb796d73d877199e4 P c09724fc20c4461981288a0c25a03743 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } }
I20240502 21:35:19.893311 25685 leader_election.cc:302] T 6b25593b6b044eebb796d73d877199e4 P c09724fc20c4461981288a0c25a03743 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c09724fc20c4461981288a0c25a03743; no voters: 
I20240502 21:35:19.894044 25685 leader_election.cc:288] T 6b25593b6b044eebb796d73d877199e4 P c09724fc20c4461981288a0c25a03743 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 21:35:19.894194 25687 raft_consensus.cc:2781] T 6b25593b6b044eebb796d73d877199e4 P c09724fc20c4461981288a0c25a03743 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:19.896134 25685 ts_tablet_manager.cc:1398] T 6b25593b6b044eebb796d73d877199e4 P c09724fc20c4461981288a0c25a03743: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.009s
I20240502 21:35:19.896739 25685 tablet_bootstrap.cc:492] T 10c2741fd1924b69a8f98ef874a85610 P c09724fc20c4461981288a0c25a03743: Bootstrap starting.
I20240502 21:35:19.897078 25687 raft_consensus.cc:686] T 6b25593b6b044eebb796d73d877199e4 P c09724fc20c4461981288a0c25a03743 [term 1 LEADER]: Becoming Leader. State: Replica: c09724fc20c4461981288a0c25a03743, State: Running, Role: LEADER
I20240502 21:35:19.897799 25687 consensus_queue.cc:237] T 6b25593b6b044eebb796d73d877199e4 P c09724fc20c4461981288a0c25a03743 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } }
I20240502 21:35:19.898473 25685 tablet_bootstrap.cc:654] T 10c2741fd1924b69a8f98ef874a85610 P c09724fc20c4461981288a0c25a03743: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:19.900863 25685 tablet_bootstrap.cc:492] T 10c2741fd1924b69a8f98ef874a85610 P c09724fc20c4461981288a0c25a03743: No bootstrap required, opened a new log
I20240502 21:35:19.901129 25685 ts_tablet_manager.cc:1367] T 10c2741fd1924b69a8f98ef874a85610 P c09724fc20c4461981288a0c25a03743: Time spent bootstrapping tablet: real 0.004s	user 0.004s	sys 0.000s
I20240502 21:35:19.902331 25685 raft_consensus.cc:348] T 10c2741fd1924b69a8f98ef874a85610 P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } }
I20240502 21:35:19.902555 25685 raft_consensus.cc:374] T 10c2741fd1924b69a8f98ef874a85610 P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:19.902705 25685 raft_consensus.cc:729] T 10c2741fd1924b69a8f98ef874a85610 P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c09724fc20c4461981288a0c25a03743, State: Initialized, Role: FOLLOWER
I20240502 21:35:19.902969 25685 consensus_queue.cc:260] T 10c2741fd1924b69a8f98ef874a85610 P c09724fc20c4461981288a0c25a03743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } }
I20240502 21:35:19.903227 25685 raft_consensus.cc:388] T 10c2741fd1924b69a8f98ef874a85610 P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:35:19.903322 25685 raft_consensus.cc:482] T 10c2741fd1924b69a8f98ef874a85610 P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:35:19.903451 25685 raft_consensus.cc:3037] T 10c2741fd1924b69a8f98ef874a85610 P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:19.904281 25051 catalog_manager.cc:5484] T 6b25593b6b044eebb796d73d877199e4 P c09724fc20c4461981288a0c25a03743 reported cstate change: term changed from 0 to 1, leader changed from <none> to c09724fc20c4461981288a0c25a03743 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "c09724fc20c4461981288a0c25a03743" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } } }
I20240502 21:35:19.907003 25685 raft_consensus.cc:504] T 10c2741fd1924b69a8f98ef874a85610 P c09724fc20c4461981288a0c25a03743 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } }
I20240502 21:35:19.907330 25685 leader_election.cc:302] T 10c2741fd1924b69a8f98ef874a85610 P c09724fc20c4461981288a0c25a03743 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c09724fc20c4461981288a0c25a03743; no voters: 
I20240502 21:35:19.907547 25685 leader_election.cc:288] T 10c2741fd1924b69a8f98ef874a85610 P c09724fc20c4461981288a0c25a03743 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 21:35:19.907629 25687 raft_consensus.cc:2781] T 10c2741fd1924b69a8f98ef874a85610 P c09724fc20c4461981288a0c25a03743 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:19.907848 25687 raft_consensus.cc:686] T 10c2741fd1924b69a8f98ef874a85610 P c09724fc20c4461981288a0c25a03743 [term 1 LEADER]: Becoming Leader. State: Replica: c09724fc20c4461981288a0c25a03743, State: Running, Role: LEADER
I20240502 21:35:19.908102 25685 ts_tablet_manager.cc:1398] T 10c2741fd1924b69a8f98ef874a85610 P c09724fc20c4461981288a0c25a03743: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.003s
I20240502 21:35:19.908185 25687 consensus_queue.cc:237] T 10c2741fd1924b69a8f98ef874a85610 P c09724fc20c4461981288a0c25a03743 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } }
I20240502 21:35:19.912897 25051 catalog_manager.cc:5484] T 10c2741fd1924b69a8f98ef874a85610 P c09724fc20c4461981288a0c25a03743 reported cstate change: term changed from 0 to 1, leader changed from <none> to c09724fc20c4461981288a0c25a03743 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "c09724fc20c4461981288a0c25a03743" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } } }
I20240502 21:35:20.528120 25051 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51332:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 21:35:20.540274 25174 tablet_service.cc:1467] Processing CreateTablet for tablet c3154ccea73a4989abc8aeaf82ad3581 (DEFAULT_TABLE table=rebalance_test_table_1 [id=68de9b13ac7a4736b711c04f581a5ea5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:20.540419 25308 tablet_service.cc:1467] Processing CreateTablet for tablet d59514c582ae4051b02317ae5f3dc57d (DEFAULT_TABLE table=rebalance_test_table_1 [id=68de9b13ac7a4736b711c04f581a5ea5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:20.540858 25174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3154ccea73a4989abc8aeaf82ad3581. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:20.540984 25308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d59514c582ae4051b02317ae5f3dc57d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:20.541337 25173 tablet_service.cc:1467] Processing CreateTablet for tablet c9b09d9af8f74895bcd9afdd5f2aece1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=68de9b13ac7a4736b711c04f581a5ea5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:20.541816 25173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9b09d9af8f74895bcd9afdd5f2aece1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:20.546757 25721 tablet_bootstrap.cc:492] T d59514c582ae4051b02317ae5f3dc57d P c09724fc20c4461981288a0c25a03743: Bootstrap starting.
I20240502 21:35:20.549221 25722 tablet_bootstrap.cc:492] T c9b09d9af8f74895bcd9afdd5f2aece1 P f638cb4a3c7f4d368fff500a0cb1e3bc: Bootstrap starting.
I20240502 21:35:20.549532 25721 tablet_bootstrap.cc:654] T d59514c582ae4051b02317ae5f3dc57d P c09724fc20c4461981288a0c25a03743: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:20.550784 25722 tablet_bootstrap.cc:654] T c9b09d9af8f74895bcd9afdd5f2aece1 P f638cb4a3c7f4d368fff500a0cb1e3bc: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:20.551664 25721 tablet_bootstrap.cc:492] T d59514c582ae4051b02317ae5f3dc57d P c09724fc20c4461981288a0c25a03743: No bootstrap required, opened a new log
I20240502 21:35:20.551828 25721 ts_tablet_manager.cc:1367] T d59514c582ae4051b02317ae5f3dc57d P c09724fc20c4461981288a0c25a03743: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 21:35:20.552707 25721 raft_consensus.cc:348] T d59514c582ae4051b02317ae5f3dc57d P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } }
I20240502 21:35:20.552872 25721 raft_consensus.cc:374] T d59514c582ae4051b02317ae5f3dc57d P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:20.552968 25721 raft_consensus.cc:729] T d59514c582ae4051b02317ae5f3dc57d P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c09724fc20c4461981288a0c25a03743, State: Initialized, Role: FOLLOWER
I20240502 21:35:20.553232 25721 consensus_queue.cc:260] T d59514c582ae4051b02317ae5f3dc57d P c09724fc20c4461981288a0c25a03743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } }
I20240502 21:35:20.553412 25721 raft_consensus.cc:388] T d59514c582ae4051b02317ae5f3dc57d P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:35:20.553530 25721 raft_consensus.cc:482] T d59514c582ae4051b02317ae5f3dc57d P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:35:20.553661 25721 raft_consensus.cc:3037] T d59514c582ae4051b02317ae5f3dc57d P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:20.556963 25722 tablet_bootstrap.cc:492] T c9b09d9af8f74895bcd9afdd5f2aece1 P f638cb4a3c7f4d368fff500a0cb1e3bc: No bootstrap required, opened a new log
I20240502 21:35:20.557166 25722 ts_tablet_manager.cc:1367] T c9b09d9af8f74895bcd9afdd5f2aece1 P f638cb4a3c7f4d368fff500a0cb1e3bc: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.003s
I20240502 21:35:20.557837 25722 raft_consensus.cc:348] T c9b09d9af8f74895bcd9afdd5f2aece1 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } }
I20240502 21:35:20.558005 25722 raft_consensus.cc:374] T c9b09d9af8f74895bcd9afdd5f2aece1 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:20.558105 25722 raft_consensus.cc:729] T c9b09d9af8f74895bcd9afdd5f2aece1 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f638cb4a3c7f4d368fff500a0cb1e3bc, State: Initialized, Role: FOLLOWER
I20240502 21:35:20.558650 25722 consensus_queue.cc:260] T c9b09d9af8f74895bcd9afdd5f2aece1 P f638cb4a3c7f4d368fff500a0cb1e3bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } }
I20240502 21:35:20.558919 25722 raft_consensus.cc:388] T c9b09d9af8f74895bcd9afdd5f2aece1 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:35:20.559011 25722 raft_consensus.cc:482] T c9b09d9af8f74895bcd9afdd5f2aece1 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:35:20.559180 25722 raft_consensus.cc:3037] T c9b09d9af8f74895bcd9afdd5f2aece1 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:20.565977 25722 raft_consensus.cc:504] T c9b09d9af8f74895bcd9afdd5f2aece1 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } }
I20240502 21:35:20.565966 25721 raft_consensus.cc:504] T d59514c582ae4051b02317ae5f3dc57d P c09724fc20c4461981288a0c25a03743 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } }
I20240502 21:35:20.566359 25721 leader_election.cc:302] T d59514c582ae4051b02317ae5f3dc57d P c09724fc20c4461981288a0c25a03743 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c09724fc20c4461981288a0c25a03743; no voters: 
I20240502 21:35:20.566359 25722 leader_election.cc:302] T c9b09d9af8f74895bcd9afdd5f2aece1 P f638cb4a3c7f4d368fff500a0cb1e3bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f638cb4a3c7f4d368fff500a0cb1e3bc; no voters: 
I20240502 21:35:20.566624 25721 leader_election.cc:288] T d59514c582ae4051b02317ae5f3dc57d P c09724fc20c4461981288a0c25a03743 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 21:35:20.566784 25715 raft_consensus.cc:2781] T d59514c582ae4051b02317ae5f3dc57d P c09724fc20c4461981288a0c25a03743 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:20.567077 25722 leader_election.cc:288] T c9b09d9af8f74895bcd9afdd5f2aece1 P f638cb4a3c7f4d368fff500a0cb1e3bc [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 21:35:20.567451 25725 raft_consensus.cc:2781] T c9b09d9af8f74895bcd9afdd5f2aece1 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:20.578163 25725 raft_consensus.cc:686] T c9b09d9af8f74895bcd9afdd5f2aece1 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 1 LEADER]: Becoming Leader. State: Replica: f638cb4a3c7f4d368fff500a0cb1e3bc, State: Running, Role: LEADER
I20240502 21:35:20.579196 25715 raft_consensus.cc:686] T d59514c582ae4051b02317ae5f3dc57d P c09724fc20c4461981288a0c25a03743 [term 1 LEADER]: Becoming Leader. State: Replica: c09724fc20c4461981288a0c25a03743, State: Running, Role: LEADER
I20240502 21:35:20.579262 25722 ts_tablet_manager.cc:1398] T c9b09d9af8f74895bcd9afdd5f2aece1 P f638cb4a3c7f4d368fff500a0cb1e3bc: Time spent starting tablet: real 0.022s	user 0.004s	sys 0.010s
I20240502 21:35:20.579615 25722 tablet_bootstrap.cc:492] T c3154ccea73a4989abc8aeaf82ad3581 P f638cb4a3c7f4d368fff500a0cb1e3bc: Bootstrap starting.
I20240502 21:35:20.578954 25725 consensus_queue.cc:237] T c9b09d9af8f74895bcd9afdd5f2aece1 P f638cb4a3c7f4d368fff500a0cb1e3bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } }
I20240502 21:35:20.579567 25715 consensus_queue.cc:237] T d59514c582ae4051b02317ae5f3dc57d P c09724fc20c4461981288a0c25a03743 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } }
I20240502 21:35:20.582381 25722 tablet_bootstrap.cc:654] T c3154ccea73a4989abc8aeaf82ad3581 P f638cb4a3c7f4d368fff500a0cb1e3bc: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:20.582888 25721 ts_tablet_manager.cc:1398] T d59514c582ae4051b02317ae5f3dc57d P c09724fc20c4461981288a0c25a03743: Time spent starting tablet: real 0.031s	user 0.004s	sys 0.016s
I20240502 21:35:20.586658 25722 tablet_bootstrap.cc:492] T c3154ccea73a4989abc8aeaf82ad3581 P f638cb4a3c7f4d368fff500a0cb1e3bc: No bootstrap required, opened a new log
I20240502 21:35:20.586360 25051 catalog_manager.cc:5484] T d59514c582ae4051b02317ae5f3dc57d P c09724fc20c4461981288a0c25a03743 reported cstate change: term changed from 0 to 1, leader changed from <none> to c09724fc20c4461981288a0c25a03743 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "c09724fc20c4461981288a0c25a03743" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } } }
I20240502 21:35:20.586863 25722 ts_tablet_manager.cc:1367] T c3154ccea73a4989abc8aeaf82ad3581 P f638cb4a3c7f4d368fff500a0cb1e3bc: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.003s
I20240502 21:35:20.587826 25722 raft_consensus.cc:348] T c3154ccea73a4989abc8aeaf82ad3581 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } }
I20240502 21:35:20.588070 25722 raft_consensus.cc:374] T c3154ccea73a4989abc8aeaf82ad3581 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:20.587900 25052 catalog_manager.cc:5484] T c9b09d9af8f74895bcd9afdd5f2aece1 P f638cb4a3c7f4d368fff500a0cb1e3bc reported cstate change: term changed from 0 to 1, leader changed from <none> to f638cb4a3c7f4d368fff500a0cb1e3bc (127.27.196.65). New cstate: current_term: 1 leader_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } } }
I20240502 21:35:20.588168 25722 raft_consensus.cc:729] T c3154ccea73a4989abc8aeaf82ad3581 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f638cb4a3c7f4d368fff500a0cb1e3bc, State: Initialized, Role: FOLLOWER
I20240502 21:35:20.588467 25722 consensus_queue.cc:260] T c3154ccea73a4989abc8aeaf82ad3581 P f638cb4a3c7f4d368fff500a0cb1e3bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } }
I20240502 21:35:20.588670 25722 raft_consensus.cc:388] T c3154ccea73a4989abc8aeaf82ad3581 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:35:20.588780 25722 raft_consensus.cc:482] T c3154ccea73a4989abc8aeaf82ad3581 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:35:20.588897 25722 raft_consensus.cc:3037] T c3154ccea73a4989abc8aeaf82ad3581 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:20.591857 25722 raft_consensus.cc:504] T c3154ccea73a4989abc8aeaf82ad3581 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } }
I20240502 21:35:20.592137 25722 leader_election.cc:302] T c3154ccea73a4989abc8aeaf82ad3581 P f638cb4a3c7f4d368fff500a0cb1e3bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f638cb4a3c7f4d368fff500a0cb1e3bc; no voters: 
I20240502 21:35:20.592442 25722 leader_election.cc:288] T c3154ccea73a4989abc8aeaf82ad3581 P f638cb4a3c7f4d368fff500a0cb1e3bc [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 21:35:20.592443 25725 raft_consensus.cc:2781] T c3154ccea73a4989abc8aeaf82ad3581 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:20.592664 25725 raft_consensus.cc:686] T c3154ccea73a4989abc8aeaf82ad3581 P f638cb4a3c7f4d368fff500a0cb1e3bc [term 1 LEADER]: Becoming Leader. State: Replica: f638cb4a3c7f4d368fff500a0cb1e3bc, State: Running, Role: LEADER
I20240502 21:35:20.592929 25725 consensus_queue.cc:237] T c3154ccea73a4989abc8aeaf82ad3581 P f638cb4a3c7f4d368fff500a0cb1e3bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } }
I20240502 21:35:20.593362 25722 ts_tablet_manager.cc:1398] T c3154ccea73a4989abc8aeaf82ad3581 P f638cb4a3c7f4d368fff500a0cb1e3bc: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20240502 21:35:20.595892 25051 catalog_manager.cc:5484] T c3154ccea73a4989abc8aeaf82ad3581 P f638cb4a3c7f4d368fff500a0cb1e3bc reported cstate change: term changed from 0 to 1, leader changed from <none> to f638cb4a3c7f4d368fff500a0cb1e3bc (127.27.196.65). New cstate: current_term: 1 leader_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } } }
I20240502 21:35:21.177212 25051 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51332:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 21:35:21.189116 25307 tablet_service.cc:1467] Processing CreateTablet for tablet e40abf89427d42eea7e89bf9ae9159ac (DEFAULT_TABLE table=rebalance_test_table_2 [id=bd233bc65ba64811bccf787779c74dbc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:21.189164 25308 tablet_service.cc:1467] Processing CreateTablet for tablet d800e9784ad74fe6ab4e85fe9440b17d (DEFAULT_TABLE table=rebalance_test_table_2 [id=bd233bc65ba64811bccf787779c74dbc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:21.189642 25307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e40abf89427d42eea7e89bf9ae9159ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:21.190454 25173 tablet_service.cc:1467] Processing CreateTablet for tablet 783f439e624a407b82923cee9d9e185b (DEFAULT_TABLE table=rebalance_test_table_2 [id=bd233bc65ba64811bccf787779c74dbc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:21.190896 25173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 783f439e624a407b82923cee9d9e185b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:21.191340 25308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d800e9784ad74fe6ab4e85fe9440b17d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:21.204387 25761 tablet_bootstrap.cc:492] T e40abf89427d42eea7e89bf9ae9159ac P c09724fc20c4461981288a0c25a03743: Bootstrap starting.
I20240502 21:35:21.205683 25762 tablet_bootstrap.cc:492] T 783f439e624a407b82923cee9d9e185b P f638cb4a3c7f4d368fff500a0cb1e3bc: Bootstrap starting.
I20240502 21:35:21.206558 25761 tablet_bootstrap.cc:654] T e40abf89427d42eea7e89bf9ae9159ac P c09724fc20c4461981288a0c25a03743: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:21.208344 25762 tablet_bootstrap.cc:654] T 783f439e624a407b82923cee9d9e185b P f638cb4a3c7f4d368fff500a0cb1e3bc: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:21.211314 25761 tablet_bootstrap.cc:492] T e40abf89427d42eea7e89bf9ae9159ac P c09724fc20c4461981288a0c25a03743: No bootstrap required, opened a new log
I20240502 21:35:21.211345 25762 tablet_bootstrap.cc:492] T 783f439e624a407b82923cee9d9e185b P f638cb4a3c7f4d368fff500a0cb1e3bc: No bootstrap required, opened a new log
I20240502 21:35:21.211519 25762 ts_tablet_manager.cc:1367] T 783f439e624a407b82923cee9d9e185b P f638cb4a3c7f4d368fff500a0cb1e3bc: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
I20240502 21:35:21.211519 25761 ts_tablet_manager.cc:1367] T e40abf89427d42eea7e89bf9ae9159ac P c09724fc20c4461981288a0c25a03743: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.006s
I20240502 21:35:21.212397 25762 raft_consensus.cc:348] T 783f439e624a407b82923cee9d9e185b P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } }
I20240502 21:35:21.212618 25762 raft_consensus.cc:374] T 783f439e624a407b82923cee9d9e185b P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:21.212734 25762 raft_consensus.cc:729] T 783f439e624a407b82923cee9d9e185b P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f638cb4a3c7f4d368fff500a0cb1e3bc, State: Initialized, Role: FOLLOWER
I20240502 21:35:21.213028 25762 consensus_queue.cc:260] T 783f439e624a407b82923cee9d9e185b P f638cb4a3c7f4d368fff500a0cb1e3bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } }
I20240502 21:35:21.213258 25762 raft_consensus.cc:388] T 783f439e624a407b82923cee9d9e185b P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:35:21.213371 25762 raft_consensus.cc:482] T 783f439e624a407b82923cee9d9e185b P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:35:21.213513 25762 raft_consensus.cc:3037] T 783f439e624a407b82923cee9d9e185b P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:21.214097 25761 raft_consensus.cc:348] T e40abf89427d42eea7e89bf9ae9159ac P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } }
I20240502 21:35:21.214295 25761 raft_consensus.cc:374] T e40abf89427d42eea7e89bf9ae9159ac P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:21.214403 25761 raft_consensus.cc:729] T e40abf89427d42eea7e89bf9ae9159ac P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c09724fc20c4461981288a0c25a03743, State: Initialized, Role: FOLLOWER
I20240502 21:35:21.214666 25761 consensus_queue.cc:260] T e40abf89427d42eea7e89bf9ae9159ac P c09724fc20c4461981288a0c25a03743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } }
I20240502 21:35:21.214861 25761 raft_consensus.cc:388] T e40abf89427d42eea7e89bf9ae9159ac P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:35:21.214955 25761 raft_consensus.cc:482] T e40abf89427d42eea7e89bf9ae9159ac P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:35:21.215071 25761 raft_consensus.cc:3037] T e40abf89427d42eea7e89bf9ae9159ac P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:21.216538 25762 raft_consensus.cc:504] T 783f439e624a407b82923cee9d9e185b P f638cb4a3c7f4d368fff500a0cb1e3bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } }
I20240502 21:35:21.216826 25762 leader_election.cc:302] T 783f439e624a407b82923cee9d9e185b P f638cb4a3c7f4d368fff500a0cb1e3bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f638cb4a3c7f4d368fff500a0cb1e3bc; no voters: 
I20240502 21:35:21.217063 25762 leader_election.cc:288] T 783f439e624a407b82923cee9d9e185b P f638cb4a3c7f4d368fff500a0cb1e3bc [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 21:35:21.217168 25725 raft_consensus.cc:2781] T 783f439e624a407b82923cee9d9e185b P f638cb4a3c7f4d368fff500a0cb1e3bc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:21.217895 25762 ts_tablet_manager.cc:1398] T 783f439e624a407b82923cee9d9e185b P f638cb4a3c7f4d368fff500a0cb1e3bc: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20240502 21:35:21.218680 25761 raft_consensus.cc:504] T e40abf89427d42eea7e89bf9ae9159ac P c09724fc20c4461981288a0c25a03743 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } }
I20240502 21:35:21.218940 25761 leader_election.cc:302] T e40abf89427d42eea7e89bf9ae9159ac P c09724fc20c4461981288a0c25a03743 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c09724fc20c4461981288a0c25a03743; no voters: 
I20240502 21:35:21.219473 25761 leader_election.cc:288] T e40abf89427d42eea7e89bf9ae9159ac P c09724fc20c4461981288a0c25a03743 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 21:35:21.219753 25767 raft_consensus.cc:2781] T e40abf89427d42eea7e89bf9ae9159ac P c09724fc20c4461981288a0c25a03743 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:21.220625 25761 ts_tablet_manager.cc:1398] T e40abf89427d42eea7e89bf9ae9159ac P c09724fc20c4461981288a0c25a03743: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.002s
I20240502 21:35:21.220702 25767 raft_consensus.cc:686] T e40abf89427d42eea7e89bf9ae9159ac P c09724fc20c4461981288a0c25a03743 [term 1 LEADER]: Becoming Leader. State: Replica: c09724fc20c4461981288a0c25a03743, State: Running, Role: LEADER
I20240502 21:35:21.220966 25761 tablet_bootstrap.cc:492] T d800e9784ad74fe6ab4e85fe9440b17d P c09724fc20c4461981288a0c25a03743: Bootstrap starting.
I20240502 21:35:21.221189 25767 consensus_queue.cc:237] T e40abf89427d42eea7e89bf9ae9159ac P c09724fc20c4461981288a0c25a03743 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } }
I20240502 21:35:21.223353 25761 tablet_bootstrap.cc:654] T d800e9784ad74fe6ab4e85fe9440b17d P c09724fc20c4461981288a0c25a03743: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:21.224565 25051 catalog_manager.cc:5484] T e40abf89427d42eea7e89bf9ae9159ac P c09724fc20c4461981288a0c25a03743 reported cstate change: term changed from 0 to 1, leader changed from <none> to c09724fc20c4461981288a0c25a03743 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "c09724fc20c4461981288a0c25a03743" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } } }
I20240502 21:35:21.226933 25761 tablet_bootstrap.cc:492] T d800e9784ad74fe6ab4e85fe9440b17d P c09724fc20c4461981288a0c25a03743: No bootstrap required, opened a new log
I20240502 21:35:21.227177 25761 ts_tablet_manager.cc:1367] T d800e9784ad74fe6ab4e85fe9440b17d P c09724fc20c4461981288a0c25a03743: Time spent bootstrapping tablet: real 0.006s	user 0.001s	sys 0.003s
I20240502 21:35:21.228086 25761 raft_consensus.cc:348] T d800e9784ad74fe6ab4e85fe9440b17d P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } }
I20240502 21:35:21.228376 25761 raft_consensus.cc:374] T d800e9784ad74fe6ab4e85fe9440b17d P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:21.228518 25761 raft_consensus.cc:729] T d800e9784ad74fe6ab4e85fe9440b17d P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c09724fc20c4461981288a0c25a03743, State: Initialized, Role: FOLLOWER
I20240502 21:35:21.228797 25761 consensus_queue.cc:260] T d800e9784ad74fe6ab4e85fe9440b17d P c09724fc20c4461981288a0c25a03743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } }
I20240502 21:35:21.229069 25761 raft_consensus.cc:388] T d800e9784ad74fe6ab4e85fe9440b17d P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:35:21.229189 25761 raft_consensus.cc:482] T d800e9784ad74fe6ab4e85fe9440b17d P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:35:21.229295 25761 raft_consensus.cc:3037] T d800e9784ad74fe6ab4e85fe9440b17d P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:21.231537 25725 raft_consensus.cc:686] T 783f439e624a407b82923cee9d9e185b P f638cb4a3c7f4d368fff500a0cb1e3bc [term 1 LEADER]: Becoming Leader. State: Replica: f638cb4a3c7f4d368fff500a0cb1e3bc, State: Running, Role: LEADER
I20240502 21:35:21.231578 25761 raft_consensus.cc:504] T d800e9784ad74fe6ab4e85fe9440b17d P c09724fc20c4461981288a0c25a03743 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } }
I20240502 21:35:21.231833 25761 leader_election.cc:302] T d800e9784ad74fe6ab4e85fe9440b17d P c09724fc20c4461981288a0c25a03743 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c09724fc20c4461981288a0c25a03743; no voters: 
I20240502 21:35:21.232015 25725 consensus_queue.cc:237] T 783f439e624a407b82923cee9d9e185b P f638cb4a3c7f4d368fff500a0cb1e3bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } }
I20240502 21:35:21.232239 25769 raft_consensus.cc:2781] T d800e9784ad74fe6ab4e85fe9440b17d P c09724fc20c4461981288a0c25a03743 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:21.232421 25769 raft_consensus.cc:686] T d800e9784ad74fe6ab4e85fe9440b17d P c09724fc20c4461981288a0c25a03743 [term 1 LEADER]: Becoming Leader. State: Replica: c09724fc20c4461981288a0c25a03743, State: Running, Role: LEADER
I20240502 21:35:21.232738 25769 consensus_queue.cc:237] T d800e9784ad74fe6ab4e85fe9440b17d P c09724fc20c4461981288a0c25a03743 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } }
I20240502 21:35:21.234119 25761 leader_election.cc:288] T d800e9784ad74fe6ab4e85fe9440b17d P c09724fc20c4461981288a0c25a03743 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 21:35:21.234750 25761 ts_tablet_manager.cc:1398] T d800e9784ad74fe6ab4e85fe9440b17d P c09724fc20c4461981288a0c25a03743: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.002s
I20240502 21:35:21.235821 25051 catalog_manager.cc:5484] T 783f439e624a407b82923cee9d9e185b P f638cb4a3c7f4d368fff500a0cb1e3bc reported cstate change: term changed from 0 to 1, leader changed from <none> to f638cb4a3c7f4d368fff500a0cb1e3bc (127.27.196.65). New cstate: current_term: 1 leader_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } } }
I20240502 21:35:21.242610 25051 catalog_manager.cc:5484] T d800e9784ad74fe6ab4e85fe9440b17d P c09724fc20c4461981288a0c25a03743 reported cstate change: term changed from 0 to 1, leader changed from <none> to c09724fc20c4461981288a0c25a03743 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "c09724fc20c4461981288a0c25a03743" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } } }
I20240502 21:35:21.820549 25051 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51332:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 21:35:21.830529 25173 tablet_service.cc:1467] Processing CreateTablet for tablet 150b2f1260884d4cbf3af7667b2c36be (DEFAULT_TABLE table=rebalance_test_table_3 [id=431ea390b5d7424ea5af62c150404294]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:21.830605 25307 tablet_service.cc:1467] Processing CreateTablet for tablet 8f48b2693ad74bc49870505a09c818e7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=431ea390b5d7424ea5af62c150404294]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:21.830857 25174 tablet_service.cc:1467] Processing CreateTablet for tablet db4ddf4f6a404f7186590e4c995939bd (DEFAULT_TABLE table=rebalance_test_table_3 [id=431ea390b5d7424ea5af62c150404294]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:21.831087 25173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 150b2f1260884d4cbf3af7667b2c36be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:21.831077 25307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f48b2693ad74bc49870505a09c818e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:21.831422 25174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db4ddf4f6a404f7186590e4c995939bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:21.838639 25801 tablet_bootstrap.cc:492] T db4ddf4f6a404f7186590e4c995939bd P f638cb4a3c7f4d368fff500a0cb1e3bc: Bootstrap starting.
I20240502 21:35:21.838686 25802 tablet_bootstrap.cc:492] T 8f48b2693ad74bc49870505a09c818e7 P c09724fc20c4461981288a0c25a03743: Bootstrap starting.
I20240502 21:35:21.840283 25802 tablet_bootstrap.cc:654] T 8f48b2693ad74bc49870505a09c818e7 P c09724fc20c4461981288a0c25a03743: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:21.840305 25801 tablet_bootstrap.cc:654] T db4ddf4f6a404f7186590e4c995939bd P f638cb4a3c7f4d368fff500a0cb1e3bc: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:21.842453 25802 tablet_bootstrap.cc:492] T 8f48b2693ad74bc49870505a09c818e7 P c09724fc20c4461981288a0c25a03743: No bootstrap required, opened a new log
I20240502 21:35:21.842638 25802 ts_tablet_manager.cc:1367] T 8f48b2693ad74bc49870505a09c818e7 P c09724fc20c4461981288a0c25a03743: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 21:35:21.842679 25801 tablet_bootstrap.cc:492] T db4ddf4f6a404f7186590e4c995939bd P f638cb4a3c7f4d368fff500a0cb1e3bc: No bootstrap required, opened a new log
I20240502 21:35:21.842813 25801 ts_tablet_manager.cc:1367] T db4ddf4f6a404f7186590e4c995939bd P f638cb4a3c7f4d368fff500a0cb1e3bc: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 21:35:21.843398 25802 raft_consensus.cc:348] T 8f48b2693ad74bc49870505a09c818e7 P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } }
I20240502 21:35:21.843485 25801 raft_consensus.cc:348] T db4ddf4f6a404f7186590e4c995939bd P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } }
I20240502 21:35:21.843587 25802 raft_consensus.cc:374] T 8f48b2693ad74bc49870505a09c818e7 P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:21.843652 25801 raft_consensus.cc:374] T db4ddf4f6a404f7186590e4c995939bd P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:21.843685 25802 raft_consensus.cc:729] T 8f48b2693ad74bc49870505a09c818e7 P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c09724fc20c4461981288a0c25a03743, State: Initialized, Role: FOLLOWER
I20240502 21:35:21.843748 25801 raft_consensus.cc:729] T db4ddf4f6a404f7186590e4c995939bd P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f638cb4a3c7f4d368fff500a0cb1e3bc, State: Initialized, Role: FOLLOWER
I20240502 21:35:21.843933 25802 consensus_queue.cc:260] T 8f48b2693ad74bc49870505a09c818e7 P c09724fc20c4461981288a0c25a03743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } }
I20240502 21:35:21.844012 25801 consensus_queue.cc:260] T db4ddf4f6a404f7186590e4c995939bd P f638cb4a3c7f4d368fff500a0cb1e3bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } }
I20240502 21:35:21.844126 25802 raft_consensus.cc:388] T 8f48b2693ad74bc49870505a09c818e7 P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:35:21.844178 25801 raft_consensus.cc:388] T db4ddf4f6a404f7186590e4c995939bd P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:35:21.844214 25802 raft_consensus.cc:482] T 8f48b2693ad74bc49870505a09c818e7 P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:35:21.844276 25801 raft_consensus.cc:482] T db4ddf4f6a404f7186590e4c995939bd P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:35:21.844317 25802 raft_consensus.cc:3037] T 8f48b2693ad74bc49870505a09c818e7 P c09724fc20c4461981288a0c25a03743 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:21.844391 25801 raft_consensus.cc:3037] T db4ddf4f6a404f7186590e4c995939bd P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:21.846772 25801 raft_consensus.cc:504] T db4ddf4f6a404f7186590e4c995939bd P f638cb4a3c7f4d368fff500a0cb1e3bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } }
I20240502 21:35:21.846926 25802 raft_consensus.cc:504] T 8f48b2693ad74bc49870505a09c818e7 P c09724fc20c4461981288a0c25a03743 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } }
I20240502 21:35:21.847061 25801 leader_election.cc:302] T db4ddf4f6a404f7186590e4c995939bd P f638cb4a3c7f4d368fff500a0cb1e3bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f638cb4a3c7f4d368fff500a0cb1e3bc; no voters: 
I20240502 21:35:21.847218 25802 leader_election.cc:302] T 8f48b2693ad74bc49870505a09c818e7 P c09724fc20c4461981288a0c25a03743 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c09724fc20c4461981288a0c25a03743; no voters: 
I20240502 21:35:21.847513 25767 raft_consensus.cc:2781] T 8f48b2693ad74bc49870505a09c818e7 P c09724fc20c4461981288a0c25a03743 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:21.847674 25801 leader_election.cc:288] T db4ddf4f6a404f7186590e4c995939bd P f638cb4a3c7f4d368fff500a0cb1e3bc [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 21:35:21.847844 25805 raft_consensus.cc:2781] T db4ddf4f6a404f7186590e4c995939bd P f638cb4a3c7f4d368fff500a0cb1e3bc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:21.847976 25767 raft_consensus.cc:686] T 8f48b2693ad74bc49870505a09c818e7 P c09724fc20c4461981288a0c25a03743 [term 1 LEADER]: Becoming Leader. State: Replica: c09724fc20c4461981288a0c25a03743, State: Running, Role: LEADER
I20240502 21:35:21.848297 25767 consensus_queue.cc:237] T 8f48b2693ad74bc49870505a09c818e7 P c09724fc20c4461981288a0c25a03743 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } }
I20240502 21:35:21.848402 25805 raft_consensus.cc:686] T db4ddf4f6a404f7186590e4c995939bd P f638cb4a3c7f4d368fff500a0cb1e3bc [term 1 LEADER]: Becoming Leader. State: Replica: f638cb4a3c7f4d368fff500a0cb1e3bc, State: Running, Role: LEADER
I20240502 21:35:21.848781 25805 consensus_queue.cc:237] T db4ddf4f6a404f7186590e4c995939bd P f638cb4a3c7f4d368fff500a0cb1e3bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } }
I20240502 21:35:21.853610 25802 leader_election.cc:288] T 8f48b2693ad74bc49870505a09c818e7 P c09724fc20c4461981288a0c25a03743 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 21:35:21.853471 25051 catalog_manager.cc:5484] T db4ddf4f6a404f7186590e4c995939bd P f638cb4a3c7f4d368fff500a0cb1e3bc reported cstate change: term changed from 0 to 1, leader changed from <none> to f638cb4a3c7f4d368fff500a0cb1e3bc (127.27.196.65). New cstate: current_term: 1 leader_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } } }
I20240502 21:35:21.855156 25801 ts_tablet_manager.cc:1398] T db4ddf4f6a404f7186590e4c995939bd P f638cb4a3c7f4d368fff500a0cb1e3bc: Time spent starting tablet: real 0.012s	user 0.012s	sys 0.000s
I20240502 21:35:21.855540 25801 tablet_bootstrap.cc:492] T 150b2f1260884d4cbf3af7667b2c36be P f638cb4a3c7f4d368fff500a0cb1e3bc: Bootstrap starting.
I20240502 21:35:21.856997 25802 ts_tablet_manager.cc:1398] T 8f48b2693ad74bc49870505a09c818e7 P c09724fc20c4461981288a0c25a03743: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
I20240502 21:35:21.857220 25801 tablet_bootstrap.cc:654] T 150b2f1260884d4cbf3af7667b2c36be P f638cb4a3c7f4d368fff500a0cb1e3bc: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:21.859792 25801 tablet_bootstrap.cc:492] T 150b2f1260884d4cbf3af7667b2c36be P f638cb4a3c7f4d368fff500a0cb1e3bc: No bootstrap required, opened a new log
I20240502 21:35:21.860002 25801 ts_tablet_manager.cc:1367] T 150b2f1260884d4cbf3af7667b2c36be P f638cb4a3c7f4d368fff500a0cb1e3bc: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 21:35:21.860401 25051 catalog_manager.cc:5484] T 8f48b2693ad74bc49870505a09c818e7 P c09724fc20c4461981288a0c25a03743 reported cstate change: term changed from 0 to 1, leader changed from <none> to c09724fc20c4461981288a0c25a03743 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "c09724fc20c4461981288a0c25a03743" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c09724fc20c4461981288a0c25a03743" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 45995 } } }
I20240502 21:35:21.860952 25801 raft_consensus.cc:348] T 150b2f1260884d4cbf3af7667b2c36be P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } }
I20240502 21:35:21.861182 25801 raft_consensus.cc:374] T 150b2f1260884d4cbf3af7667b2c36be P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:21.861289 25801 raft_consensus.cc:729] T 150b2f1260884d4cbf3af7667b2c36be P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f638cb4a3c7f4d368fff500a0cb1e3bc, State: Initialized, Role: FOLLOWER
I20240502 21:35:21.861575 25801 consensus_queue.cc:260] T 150b2f1260884d4cbf3af7667b2c36be P f638cb4a3c7f4d368fff500a0cb1e3bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } }
I20240502 21:35:21.861771 25801 raft_consensus.cc:388] T 150b2f1260884d4cbf3af7667b2c36be P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:35:21.861896 25801 raft_consensus.cc:482] T 150b2f1260884d4cbf3af7667b2c36be P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:35:21.862018 25801 raft_consensus.cc:3037] T 150b2f1260884d4cbf3af7667b2c36be P f638cb4a3c7f4d368fff500a0cb1e3bc [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:21.864894 25801 raft_consensus.cc:504] T 150b2f1260884d4cbf3af7667b2c36be P f638cb4a3c7f4d368fff500a0cb1e3bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } }
I20240502 21:35:21.865135 25801 leader_election.cc:302] T 150b2f1260884d4cbf3af7667b2c36be P f638cb4a3c7f4d368fff500a0cb1e3bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f638cb4a3c7f4d368fff500a0cb1e3bc; no voters: 
I20240502 21:35:21.865372 25801 leader_election.cc:288] T 150b2f1260884d4cbf3af7667b2c36be P f638cb4a3c7f4d368fff500a0cb1e3bc [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 21:35:21.865443 25808 raft_consensus.cc:2781] T 150b2f1260884d4cbf3af7667b2c36be P f638cb4a3c7f4d368fff500a0cb1e3bc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:21.865713 25808 raft_consensus.cc:686] T 150b2f1260884d4cbf3af7667b2c36be P f638cb4a3c7f4d368fff500a0cb1e3bc [term 1 LEADER]: Becoming Leader. State: Replica: f638cb4a3c7f4d368fff500a0cb1e3bc, State: Running, Role: LEADER
I20240502 21:35:21.866055 25808 consensus_queue.cc:237] T 150b2f1260884d4cbf3af7667b2c36be P f638cb4a3c7f4d368fff500a0cb1e3bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } }
I20240502 21:35:21.867159 25801 ts_tablet_manager.cc:1398] T 150b2f1260884d4cbf3af7667b2c36be P f638cb4a3c7f4d368fff500a0cb1e3bc: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.006s
I20240502 21:35:21.872123 25051 catalog_manager.cc:5484] T 150b2f1260884d4cbf3af7667b2c36be P f638cb4a3c7f4d368fff500a0cb1e3bc reported cstate change: term changed from 0 to 1, leader changed from <none> to f638cb4a3c7f4d368fff500a0cb1e3bc (127.27.196.65). New cstate: current_term: 1 leader_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f638cb4a3c7f4d368fff500a0cb1e3bc" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 32813 } } }
I20240502 21:35:22.460669 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.67:33249
--local_ip_for_outbound_sockets=127.27.196.67
--tserver_master_addrs=127.27.196.126:43485
--webserver_port=41135
--webserver_interface=127.27.196.67
--builtin_ntp_servers=127.27.196.84:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 21:35:22.874231 25835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:35:22.874559 25835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:35:22.874706 25835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:35:22.883219 25835 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:35:22.883392 25835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:35:22.883574 25835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.67
I20240502 21:35:22.893631 25835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.67:33249
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.27.196.67
--webserver_port=41135
--tserver_master_addrs=127.27.196.126:43485
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.67
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:35:22.895207 25835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:35:22.897171 25835 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:35:22.911456 25840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:22.911875 25841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:22.912020 25843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:35:22.912520 25835 server_base.cc:1034] running on GCE node
I20240502 21:35:22.913216 25835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:35:22.914378 25835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:35:22.915627 25835 hybrid_clock.cc:648] HybridClock initialized: now 1714685722915581 us; error 39 us; skew 500 ppm
I20240502 21:35:22.915957 25835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:35:22.918568 25835 webserver.cc:460] Webserver started at http://127.27.196.67:41135/ using document root <none> and password file <none>
I20240502 21:35:22.919407 25835 fs_manager.cc:359] Metadata directory not provided
I20240502 21:35:22.919536 25835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:35:22.926352 25835 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.006s	sys 0.000s
I20240502 21:35:22.929562 25849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:22.931207 25835 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240502 21:35:22.931391 25835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9190116ee076433daec7b155ae8e5e05"
format_stamp: "Formatted at 2024-05-02 21:35:14 on dist-test-slave-3bs1"
I20240502 21:35:22.932296 25835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:35:22.955001 25835 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:35:22.956063 25835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:35:22.956353 25835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:35:22.957356 25835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:35:22.959007 25835 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:35:22.959095 25835 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:22.959249 25835 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:35:22.959302 25835 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:22.998041 25835 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.67:33249
I20240502 21:35:22.998072 25964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.67:33249 every 8 connection(s)
I20240502 21:35:22.999697 25835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:35:23.007236 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 25835
I20240502 21:35:23.008199 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.68:35175
--local_ip_for_outbound_sockets=127.27.196.68
--tserver_master_addrs=127.27.196.126:43485
--webserver_port=33919
--webserver_interface=127.27.196.68
--builtin_ntp_servers=127.27.196.84:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 21:35:23.014453 25965 heartbeater.cc:347] Connected to a master server at 127.27.196.126:43485
I20240502 21:35:23.014879 25965 heartbeater.cc:464] Registering TS with master...
I20240502 21:35:23.015954 25965 heartbeater.cc:511] Master 127.27.196.126:43485 requested a full tablet report, sending...
I20240502 21:35:23.017830 25051 ts_manager.cc:194] Re-registered known tserver with Master: 9190116ee076433daec7b155ae8e5e05 (127.27.196.67:33249)
I20240502 21:35:23.018644 25051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.67:42453
W20240502 21:35:23.400815 25969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:35:23.401149 25969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:35:23.401283 25969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:35:23.409302 25969 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 21:35:23.409466 25969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:35:23.409642 25969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.68
I20240502 21:35:23.419464 25969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.68:35175
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.27.196.68
--webserver_port=33919
--tserver_master_addrs=127.27.196.126:43485
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.68
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:35:23.420873 25969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:35:23.422729 25969 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:35:23.435516 25974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:23.436204 25975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:23.436679 25977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:35:23.437330 25969 server_base.cc:1034] running on GCE node
I20240502 21:35:23.438045 25969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:35:23.439186 25969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:35:23.440460 25969 hybrid_clock.cc:648] HybridClock initialized: now 1714685723440403 us; error 53 us; skew 500 ppm
I20240502 21:35:23.440830 25969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:35:23.443444 25969 webserver.cc:460] Webserver started at http://127.27.196.68:33919/ using document root <none> and password file <none>
I20240502 21:35:23.444250 25969 fs_manager.cc:359] Metadata directory not provided
I20240502 21:35:23.444408 25969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:35:23.450202 25969 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.001s	sys 0.004s
I20240502 21:35:23.453300 25983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:23.454732 25969 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240502 21:35:23.454903 25969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "0b98de82262a426980c54b3b294dbcb5"
format_stamp: "Formatted at 2024-05-02 21:35:15 on dist-test-slave-3bs1"
I20240502 21:35:23.455859 25969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:35:23.473245 25969 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:35:23.474130 25969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:35:23.474412 25969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:35:23.475476 25969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:35:23.477192 25969 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:35:23.477270 25969 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:23.477384 25969 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:35:23.477437 25969 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:23.518726 25969 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.68:35175
I20240502 21:35:23.518772 26099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.68:35175 every 8 connection(s)
I20240502 21:35:23.520408 25969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:35:23.523933 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 25969
I20240502 21:35:23.535929 26100 heartbeater.cc:347] Connected to a master server at 127.27.196.126:43485
I20240502 21:35:23.536290 26100 heartbeater.cc:464] Registering TS with master...
I20240502 21:35:23.537276 26100 heartbeater.cc:511] Master 127.27.196.126:43485 requested a full tablet report, sending...
I20240502 21:35:23.539093 25051 ts_manager.cc:194] Re-registered known tserver with Master: 0b98de82262a426980c54b3b294dbcb5 (127.27.196.68:35175)
I20240502 21:35:23.540410 25051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.68:47527
I20240502 21:35:24.022087 25965 heartbeater.cc:503] Master 127.27.196.126:43485 was elected leader, sending a full tablet report...
I20240502 21:35:24.098008 25307 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:24.098165 26031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:35:24.103494 25898 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:35:24.107844 25174 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:35:24.315676 26031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:35:24.318631 25898 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:35:24.320122 25307 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:24.327944 25174 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:35:24.539757 26031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:35:24.540124 25898 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:35:24.543360 26100 heartbeater.cc:503] Master 127.27.196.126:43485 was elected leader, sending a full tablet report...
I20240502 21:35:24.551977 25307 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:24.559469 25174 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:35:24.853669 25898 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:35:24.853654 26031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:35:24.857290 25307 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:35:24.873064 25174 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 cdc7fe5ec4d94cb4b14b96d243b6362c | 127.27.196.126:43485 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                               Value                                                                                                |         Tags          |         Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_ca_key_size                         | 768                                                                                                                                                                                                | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                                                | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                                               | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                                                  | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | false                                                                                                                                                                                              | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                                               | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                                               | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                                                 | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                                                | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |        Value        |         Master
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.27.196.84:45969 | all 1 server(s) checked
 time_source         | builtin             | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 0b98de82262a426980c54b3b294dbcb5 | 127.27.196.68:35175 | HEALTHY | <none>   |       0        |       0
 9190116ee076433daec7b155ae8e5e05 | 127.27.196.67:33249 | HEALTHY | <none>   |       0        |       0
 c09724fc20c4461981288a0c25a03743 | 127.27.196.66:45995 | HEALTHY | <none>   |       6        |       0
 f638cb4a3c7f4d368fff500a0cb1e3bc | 127.27.196.65:32813 | HEALTHY | <none>   |       7        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
                   Flag                   |                                                                                             Value                                                                                              |         Tags          |      Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                                                            | experimental          | all 4 server(s) checked
 local_ip_for_outbound_sockets            | 127.27.196.65                                                                                                                                                                                  | experimental          | 127.27.196.65:32813
 local_ip_for_outbound_sockets            | 127.27.196.66                                                                                                                                                                                  | experimental          | 127.27.196.66:45995
 local_ip_for_outbound_sockets            | 127.27.196.67                                                                                                                                                                                  | experimental          | 127.27.196.67:33249
 local_ip_for_outbound_sockets            | 127.27.196.68                                                                                                                                                                                  | experimental          | 127.27.196.68:35175
 never_fsync                              | true                                                                                                                                                                                           | unsafe,advanced       | all 4 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                                              | unsafe,hidden         | all 4 server(s) checked
 raft_prepare_replacement_before_eviction | false                                                                                                                                                                                          | experimental,advanced | all 4 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                                           | unsafe                | all 4 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                                             | hidden                | all 4 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.27.196.65:32813
 server_dump_info_path                    | /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.27.196.66:45995
 server_dump_info_path                    | /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.27.196.67:33249
 server_dump_info_path                    | /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.27.196.68:35175

Flags of checked categories for Tablet Server:
        Flag         |        Value        |      Tablet Server
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.27.196.84:45969 | all 4 server(s) checked
 time_source         | builtin             | all 4 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 1  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 0
 First Quartile | 0
 Median         | 6
 Third Quartile | 7
 Maximum        | 7

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 5
 Tablets        | 13
 Replicas       | 13

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240502 21:35:24.938024 28433 log_verifier.cc:126] Checking tablet 10c2741fd1924b69a8f98ef874a85610
I20240502 21:35:24.947171 28433 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 10c2741fd1924b69a8f98ef874a85610
I20240502 21:35:24.947294 28433 log_verifier.cc:126] Checking tablet 150b2f1260884d4cbf3af7667b2c36be
I20240502 21:35:24.955737 28433 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 150b2f1260884d4cbf3af7667b2c36be
I20240502 21:35:24.955842 28433 log_verifier.cc:126] Checking tablet 6b25593b6b044eebb796d73d877199e4
I20240502 21:35:24.964104 28433 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 6b25593b6b044eebb796d73d877199e4
I20240502 21:35:24.964206 28433 log_verifier.cc:126] Checking tablet 783f439e624a407b82923cee9d9e185b
I20240502 21:35:24.972409 28433 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 783f439e624a407b82923cee9d9e185b
I20240502 21:35:24.972509 28433 log_verifier.cc:126] Checking tablet 8f48b2693ad74bc49870505a09c818e7
I20240502 21:35:24.982378 28433 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 8f48b2693ad74bc49870505a09c818e7
I20240502 21:35:24.982476 28433 log_verifier.cc:126] Checking tablet c3154ccea73a4989abc8aeaf82ad3581
I20240502 21:35:24.990749 28433 log_verifier.cc:177] Verified matching terms for 2 ops in tablet c3154ccea73a4989abc8aeaf82ad3581
I20240502 21:35:24.990856 28433 log_verifier.cc:126] Checking tablet c9b09d9af8f74895bcd9afdd5f2aece1
I20240502 21:35:24.999015 28433 log_verifier.cc:177] Verified matching terms for 1 ops in tablet c9b09d9af8f74895bcd9afdd5f2aece1
I20240502 21:35:24.999147 28433 log_verifier.cc:126] Checking tablet d59514c582ae4051b02317ae5f3dc57d
I20240502 21:35:25.007273 28433 log_verifier.cc:177] Verified matching terms for 1 ops in tablet d59514c582ae4051b02317ae5f3dc57d
I20240502 21:35:25.007382 28433 log_verifier.cc:126] Checking tablet d800e9784ad74fe6ab4e85fe9440b17d
I20240502 21:35:25.016034 28433 log_verifier.cc:177] Verified matching terms for 2 ops in tablet d800e9784ad74fe6ab4e85fe9440b17d
I20240502 21:35:25.016145 28433 log_verifier.cc:126] Checking tablet db4ddf4f6a404f7186590e4c995939bd
I20240502 21:35:25.024427 28433 log_verifier.cc:177] Verified matching terms for 1 ops in tablet db4ddf4f6a404f7186590e4c995939bd
I20240502 21:35:25.024535 28433 log_verifier.cc:126] Checking tablet e40abf89427d42eea7e89bf9ae9159ac
I20240502 21:35:25.032976 28433 log_verifier.cc:177] Verified matching terms for 1 ops in tablet e40abf89427d42eea7e89bf9ae9159ac
I20240502 21:35:25.033093 28433 log_verifier.cc:126] Checking tablet f2229e07dab648ba8126efd4e470ae61
I20240502 21:35:25.042079 28433 log_verifier.cc:177] Verified matching terms for 1 ops in tablet f2229e07dab648ba8126efd4e470ae61
I20240502 21:35:25.042197 28433 log_verifier.cc:126] Checking tablet f37a95aae77446a7a765afc57f5b5732
I20240502 21:35:25.050472 28433 log_verifier.cc:177] Verified matching terms for 1 ops in tablet f37a95aae77446a7a765afc57f5b5732
I20240502 21:35:25.051545 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 25112
I20240502 21:35:25.052865 25236 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:35:25.162483 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 25112
I20240502 21:35:25.186699 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 25246
I20240502 21:35:25.187899 25369 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:35:25.300776 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 25246
I20240502 21:35:25.325230 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 25835
I20240502 21:35:25.332064 25959 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:35:25.426409 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 25835
I20240502 21:35:25.449491 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 25969
I20240502 21:35:25.458735 26094 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:35:25.561621 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 25969
I20240502 21:35:25.584048 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 25021
I20240502 21:35:25.585300 25083 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:35:25.720286 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 25021
2024-05-02T21:35:25Z chronyd exiting
[       OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/0 (13210 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (13210 ms total)

[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN      ] LocationAwareRebalancingParamTest.Rebalance/idx2_rf3_t3_l_3_2
I20240502 21:35:25.790088 28433 test_util.cc:268] Using random seed: 1637240680
I20240502 21:35:25.792342 28433 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:35:25.792428 28433 ts_itest-base.cc:113] --------------
I20240502 21:35:25.792464 28433 ts_itest-base.cc:114] 5 tablet servers
I20240502 21:35:25.792500 28433 ts_itest-base.cc:115] 3 replicas per TS
I20240502 21:35:25.792544 28433 ts_itest-base.cc:116] --------------
2024-05-02T21:35:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:35:25Z Disabled control of system clock
I20240502 21:35:25.865027 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.196.126:36809
--webserver_interface=127.27.196.126
--webserver_port=0
--builtin_ntp_servers=127.27.196.84:36263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.196.126:36809
--location_mapping_cmd=/tmp/dist-test-taskBPlvDh/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240502 21:35:26.254083 26256 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 21:35:26.254499 26256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:35:26.254653 26256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:35:26.254799 26256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:35:26.263485 26256 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:35:26.263634 26256 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:35:26.263711 26256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:35:26.263971 26256 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:35:26.264108 26256 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:35:26.274390 26256 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:36263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskBPlvDh/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--ipki_ca_key_size=768
--master_addresses=127.27.196.126:36809
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.196.126:36809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.27.196.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:35:26.276087 26256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:35:26.278060 26256 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:35:26.292199 26263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:26.292156 26262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:26.292606 26265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:35:26.293134 26256 server_base.cc:1034] running on GCE node
I20240502 21:35:26.294009 26256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:35:26.295611 26256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:35:26.296880 26256 hybrid_clock.cc:648] HybridClock initialized: now 1714685726296801 us; error 51 us; skew 500 ppm
I20240502 21:35:26.297222 26256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:35:26.303817 26256 webserver.cc:460] Webserver started at http://127.27.196.126:35287/ using document root <none> and password file <none>
I20240502 21:35:26.304692 26256 fs_manager.cc:359] Metadata directory not provided
I20240502 21:35:26.304826 26256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:35:26.305130 26256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:35:26.308235 26256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "fbb16f6b4f3c4b5eb7f5c11e1c550e4c"
format_stamp: "Formatted at 2024-05-02 21:35:26 on dist-test-slave-3bs1"
I20240502 21:35:26.309098 26256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "fbb16f6b4f3c4b5eb7f5c11e1c550e4c"
format_stamp: "Formatted at 2024-05-02 21:35:26 on dist-test-slave-3bs1"
I20240502 21:35:26.315038 26256 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.003s	sys 0.004s
I20240502 21:35:26.318862 26271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:26.320446 26256 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 21:35:26.320636 26256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
uuid: "fbb16f6b4f3c4b5eb7f5c11e1c550e4c"
format_stamp: "Formatted at 2024-05-02 21:35:26 on dist-test-slave-3bs1"
I20240502 21:35:26.320839 26256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:35:26.343605 26256 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:35:26.344527 26256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:35:26.344806 26256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:35:26.366611 26256 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.126:36809
I20240502 21:35:26.366670 26322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.126:36809 every 8 connection(s)
I20240502 21:35:26.368268 26256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 21:35:26.373997 26323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:26.378512 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 26256
I20240502 21:35:26.378885 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:35:26.389931 26323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbb16f6b4f3c4b5eb7f5c11e1c550e4c: Bootstrap starting.
I20240502 21:35:26.395182 26323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbb16f6b4f3c4b5eb7f5c11e1c550e4c: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:26.397342 26323 log.cc:824] T 00000000000000000000000000000000 P fbb16f6b4f3c4b5eb7f5c11e1c550e4c: Log is configured to *not* fsync() on all Append() calls
I20240502 21:35:26.400871 26323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbb16f6b4f3c4b5eb7f5c11e1c550e4c: No bootstrap required, opened a new log
I20240502 21:35:26.408098 26323 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbb16f6b4f3c4b5eb7f5c11e1c550e4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb16f6b4f3c4b5eb7f5c11e1c550e4c" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 36809 } }
I20240502 21:35:26.408553 26323 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbb16f6b4f3c4b5eb7f5c11e1c550e4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:26.408643 26323 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbb16f6b4f3c4b5eb7f5c11e1c550e4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb16f6b4f3c4b5eb7f5c11e1c550e4c, State: Initialized, Role: FOLLOWER
I20240502 21:35:26.409330 26323 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbb16f6b4f3c4b5eb7f5c11e1c550e4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb16f6b4f3c4b5eb7f5c11e1c550e4c" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 36809 } }
I20240502 21:35:26.409561 26323 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbb16f6b4f3c4b5eb7f5c11e1c550e4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:35:26.409685 26323 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbb16f6b4f3c4b5eb7f5c11e1c550e4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:35:26.409828 26323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbb16f6b4f3c4b5eb7f5c11e1c550e4c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:26.412149 26323 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbb16f6b4f3c4b5eb7f5c11e1c550e4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb16f6b4f3c4b5eb7f5c11e1c550e4c" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 36809 } }
I20240502 21:35:26.412724 26323 leader_election.cc:302] T 00000000000000000000000000000000 P fbb16f6b4f3c4b5eb7f5c11e1c550e4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbb16f6b4f3c4b5eb7f5c11e1c550e4c; no voters: 
I20240502 21:35:26.413874 26323 leader_election.cc:288] T 00000000000000000000000000000000 P fbb16f6b4f3c4b5eb7f5c11e1c550e4c [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 21:35:26.414052 26328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbb16f6b4f3c4b5eb7f5c11e1c550e4c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:26.415045 26328 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbb16f6b4f3c4b5eb7f5c11e1c550e4c [term 1 LEADER]: Becoming Leader. State: Replica: fbb16f6b4f3c4b5eb7f5c11e1c550e4c, State: Running, Role: LEADER
I20240502 21:35:26.415710 26328 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbb16f6b4f3c4b5eb7f5c11e1c550e4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb16f6b4f3c4b5eb7f5c11e1c550e4c" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 36809 } }
I20240502 21:35:26.416261 26323 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbb16f6b4f3c4b5eb7f5c11e1c550e4c [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:35:26.418972 26330 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbb16f6b4f3c4b5eb7f5c11e1c550e4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbb16f6b4f3c4b5eb7f5c11e1c550e4c. Latest consensus state: current_term: 1 leader_uuid: "fbb16f6b4f3c4b5eb7f5c11e1c550e4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb16f6b4f3c4b5eb7f5c11e1c550e4c" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 36809 } } }
I20240502 21:35:26.419368 26330 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbb16f6b4f3c4b5eb7f5c11e1c550e4c [sys.catalog]: This master's current role is: LEADER
I20240502 21:35:26.418906 26329 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbb16f6b4f3c4b5eb7f5c11e1c550e4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbb16f6b4f3c4b5eb7f5c11e1c550e4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb16f6b4f3c4b5eb7f5c11e1c550e4c" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 36809 } } }
I20240502 21:35:26.419523 26329 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbb16f6b4f3c4b5eb7f5c11e1c550e4c [sys.catalog]: This master's current role is: LEADER
I20240502 21:35:26.421742 26335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:35:26.429873 26335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:35:26.441320 26335 catalog_manager.cc:1347] Generated new cluster ID: 8a424d7b81b04e0d85c878d7ee49e484
I20240502 21:35:26.441501 26335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:35:26.461401 26335 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:35:26.462770 26335 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:35:26.473222 26335 catalog_manager.cc:5857] T 00000000000000000000000000000000 P fbb16f6b4f3c4b5eb7f5c11e1c550e4c: Generated new TSK 0
I20240502 21:35:26.474454 26335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:35:26.490931 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.65:0
--local_ip_for_outbound_sockets=127.27.196.65
--webserver_interface=127.27.196.65
--webserver_port=0
--tserver_master_addrs=127.27.196.126:36809
--builtin_ntp_servers=127.27.196.84:36263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:35:26.911175 26347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:35:26.911562 26347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:35:26.911700 26347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:35:26.921701 26347 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:35:26.921886 26347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:35:26.922092 26347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.65
I20240502 21:35:26.933070 26347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:36263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.27.196.65
--webserver_port=0
--tserver_master_addrs=127.27.196.126:36809
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.65
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:35:26.934574 26347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:35:26.937300 26347 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:35:26.953786 26355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:26.953619 26353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:26.953639 26352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:35:26.954336 26347 server_base.cc:1034] running on GCE node
I20240502 21:35:26.955225 26347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:35:26.956338 26347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:35:26.957607 26347 hybrid_clock.cc:648] HybridClock initialized: now 1714685726957549 us; error 43 us; skew 500 ppm
I20240502 21:35:26.957998 26347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:35:26.961220 26347 webserver.cc:460] Webserver started at http://127.27.196.65:38949/ using document root <none> and password file <none>
I20240502 21:35:26.962249 26347 fs_manager.cc:359] Metadata directory not provided
I20240502 21:35:26.962441 26347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:35:26.962823 26347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:35:26.965931 26347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "cb27977c77564d38afe6e64c816eacd0"
format_stamp: "Formatted at 2024-05-02 21:35:26 on dist-test-slave-3bs1"
I20240502 21:35:26.966849 26347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "cb27977c77564d38afe6e64c816eacd0"
format_stamp: "Formatted at 2024-05-02 21:35:26 on dist-test-slave-3bs1"
I20240502 21:35:26.973865 26347 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.005s	sys 0.000s
I20240502 21:35:26.983831 26361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:26.985980 26347 fs_manager.cc:725] Time spent opening block manager: real 0.009s	user 0.000s	sys 0.002s
I20240502 21:35:26.986269 26347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cb27977c77564d38afe6e64c816eacd0"
format_stamp: "Formatted at 2024-05-02 21:35:26 on dist-test-slave-3bs1"
I20240502 21:35:26.986598 26347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:35:27.006609 26347 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:35:27.007727 26347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:35:27.008085 26347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:35:27.009250 26347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:35:27.011193 26347 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:35:27.011329 26347 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:27.011458 26347 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:35:27.011526 26347 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:27.057065 26347 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.65:36209
I20240502 21:35:27.057122 26475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.65:36209 every 8 connection(s)
I20240502 21:35:27.058714 26347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:35:27.064188 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 26347
I20240502 21:35:27.064556 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:35:27.070133 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.66:0
--local_ip_for_outbound_sockets=127.27.196.66
--webserver_interface=127.27.196.66
--webserver_port=0
--tserver_master_addrs=127.27.196.126:36809
--builtin_ntp_servers=127.27.196.84:36263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:35:27.077699 26476 heartbeater.cc:347] Connected to a master server at 127.27.196.126:36809
I20240502 21:35:27.078140 26476 heartbeater.cc:464] Registering TS with master...
I20240502 21:35:27.079450 26476 heartbeater.cc:511] Master 127.27.196.126:36809 requested a full tablet report, sending...
I20240502 21:35:27.137319 26288 ts_manager.cc:194] Registered new tserver with Master: cb27977c77564d38afe6e64c816eacd0 (127.27.196.65:36209)
I20240502 21:35:27.140123 26288 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.65:56885
W20240502 21:35:27.483759 26480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:35:27.484102 26480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:35:27.484201 26480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:35:27.492764 26480 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:35:27.492985 26480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:35:27.493193 26480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.66
I20240502 21:35:27.505196 26480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:36263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.27.196.66
--webserver_port=0
--tserver_master_addrs=127.27.196.126:36809
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.66
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:35:27.506696 26480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:35:27.508708 26480 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:35:27.523310 26487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:27.523937 26490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:27.524469 26488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:35:27.524729 26480 server_base.cc:1034] running on GCE node
I20240502 21:35:27.525985 26480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:35:27.527626 26480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:35:27.528949 26480 hybrid_clock.cc:648] HybridClock initialized: now 1714685727528881 us; error 54 us; skew 500 ppm
I20240502 21:35:27.529481 26480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:35:27.533334 26480 webserver.cc:460] Webserver started at http://127.27.196.66:44471/ using document root <none> and password file <none>
I20240502 21:35:27.534272 26480 fs_manager.cc:359] Metadata directory not provided
I20240502 21:35:27.534438 26480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:35:27.534807 26480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:35:27.537865 26480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "5b1043ab39c04568814296d029600092"
format_stamp: "Formatted at 2024-05-02 21:35:27 on dist-test-slave-3bs1"
I20240502 21:35:27.538844 26480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "5b1043ab39c04568814296d029600092"
format_stamp: "Formatted at 2024-05-02 21:35:27 on dist-test-slave-3bs1"
I20240502 21:35:27.547091 26480 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.005s	sys 0.001s
I20240502 21:35:27.552098 26496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:27.553815 26480 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
I20240502 21:35:27.554009 26480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "5b1043ab39c04568814296d029600092"
format_stamp: "Formatted at 2024-05-02 21:35:27 on dist-test-slave-3bs1"
I20240502 21:35:27.554255 26480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:35:27.592144 26480 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:35:27.593168 26480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:35:27.593509 26480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:35:27.594756 26480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:35:27.596568 26480 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:35:27.596690 26480 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:27.596822 26480 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:35:27.596909 26480 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:27.638079 26480 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.66:41285
I20240502 21:35:27.638231 26610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.66:41285 every 8 connection(s)
I20240502 21:35:27.640262 26480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:35:27.642346 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 26480
I20240502 21:35:27.642657 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:35:27.647619 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.67:0
--local_ip_for_outbound_sockets=127.27.196.67
--webserver_interface=127.27.196.67
--webserver_port=0
--tserver_master_addrs=127.27.196.126:36809
--builtin_ntp_servers=127.27.196.84:36263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:35:27.657088 26611 heartbeater.cc:347] Connected to a master server at 127.27.196.126:36809
I20240502 21:35:27.657572 26611 heartbeater.cc:464] Registering TS with master...
I20240502 21:35:27.659466 26611 heartbeater.cc:511] Master 127.27.196.126:36809 requested a full tablet report, sending...
I20240502 21:35:27.736632 26288 ts_manager.cc:194] Registered new tserver with Master: 5b1043ab39c04568814296d029600092 (127.27.196.66:41285)
I20240502 21:35:27.737929 26288 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.66:41189
W20240502 21:35:28.069351 26615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:35:28.069720 26615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:35:28.069862 26615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:35:28.077976 26615 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:35:28.078136 26615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:35:28.078306 26615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.67
I20240502 21:35:28.088145 26615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:36263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.27.196.67
--webserver_port=0
--tserver_master_addrs=127.27.196.126:36809
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.67
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:35:28.089641 26615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:35:28.091745 26615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:35:28.105366 26621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:28.105921 26624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:28.106459 26622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:35:28.106788 26615 server_base.cc:1034] running on GCE node
I20240502 21:35:28.107568 26615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:35:28.108685 26615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:35:28.109958 26615 hybrid_clock.cc:648] HybridClock initialized: now 1714685728109909 us; error 35 us; skew 500 ppm
I20240502 21:35:28.110348 26615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:35:28.113155 26615 webserver.cc:460] Webserver started at http://127.27.196.67:38257/ using document root <none> and password file <none>
I20240502 21:35:28.113971 26615 fs_manager.cc:359] Metadata directory not provided
I20240502 21:35:28.114086 26615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:35:28.114457 26615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:35:28.117630 26615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "63266faf827046629e39afa765b144bc"
format_stamp: "Formatted at 2024-05-02 21:35:28 on dist-test-slave-3bs1"
I20240502 21:35:28.118521 26615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "63266faf827046629e39afa765b144bc"
format_stamp: "Formatted at 2024-05-02 21:35:28 on dist-test-slave-3bs1"
I20240502 21:35:28.124796 26615 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.008s	sys 0.000s
I20240502 21:35:28.129266 26630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:28.131009 26615 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.000s
I20240502 21:35:28.131304 26615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "63266faf827046629e39afa765b144bc"
format_stamp: "Formatted at 2024-05-02 21:35:28 on dist-test-slave-3bs1"
I20240502 21:35:28.131572 26615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:35:28.144515 26476 heartbeater.cc:503] Master 127.27.196.126:36809 was elected leader, sending a full tablet report...
I20240502 21:35:28.149587 26615 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:35:28.150588 26615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:35:28.150898 26615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:35:28.152392 26615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:35:28.154278 26615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:35:28.154395 26615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:28.154522 26615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:35:28.154598 26615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:28.192016 26615 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.67:39203
I20240502 21:35:28.192098 26744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.67:39203 every 8 connection(s)
I20240502 21:35:28.193624 26615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:35:28.203809 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 26615
I20240502 21:35:28.204094 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:35:28.208407 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.68:0
--local_ip_for_outbound_sockets=127.27.196.68
--webserver_interface=127.27.196.68
--webserver_port=0
--tserver_master_addrs=127.27.196.126:36809
--builtin_ntp_servers=127.27.196.84:36263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:35:28.211573 26745 heartbeater.cc:347] Connected to a master server at 127.27.196.126:36809
I20240502 21:35:28.211970 26745 heartbeater.cc:464] Registering TS with master...
I20240502 21:35:28.213151 26745 heartbeater.cc:511] Master 127.27.196.126:36809 requested a full tablet report, sending...
I20240502 21:35:28.288630 26288 ts_manager.cc:194] Registered new tserver with Master: 63266faf827046629e39afa765b144bc (127.27.196.67:39203)
I20240502 21:35:28.289741 26288 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.67:33067
W20240502 21:35:28.628185 26750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:35:28.628671 26750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:35:28.628845 26750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:35:28.645428 26750 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:35:28.645627 26750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:35:28.645884 26750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.68
I20240502 21:35:28.662238 26750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:36263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.27.196.68
--webserver_port=0
--tserver_master_addrs=127.27.196.126:36809
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.68
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:35:28.663915 26750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:35:28.666675 26750 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:35:28.682718 26755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:28.683993 26756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:28.684953 26758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:35:28.686019 26750 server_base.cc:1034] running on GCE node
I20240502 21:35:28.686956 26750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:35:28.688514 26750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:35:28.689816 26750 hybrid_clock.cc:648] HybridClock initialized: now 1714685728689735 us; error 66 us; skew 500 ppm
I20240502 21:35:28.690335 26750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:35:28.693405 26750 webserver.cc:460] Webserver started at http://127.27.196.68:40663/ using document root <none> and password file <none>
I20240502 21:35:28.694274 26750 fs_manager.cc:359] Metadata directory not provided
I20240502 21:35:28.694411 26750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:35:28.694787 26750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:35:28.697925 26750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "27be1297e75b4e20ba297e7332a98f92"
format_stamp: "Formatted at 2024-05-02 21:35:28 on dist-test-slave-3bs1"
I20240502 21:35:28.698819 26750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "27be1297e75b4e20ba297e7332a98f92"
format_stamp: "Formatted at 2024-05-02 21:35:28 on dist-test-slave-3bs1"
I20240502 21:35:28.705828 26750 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.004s	sys 0.000s
I20240502 21:35:28.710403 26764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:28.712157 26750 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.002s
I20240502 21:35:28.712365 26750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "27be1297e75b4e20ba297e7332a98f92"
format_stamp: "Formatted at 2024-05-02 21:35:28 on dist-test-slave-3bs1"
I20240502 21:35:28.712622 26750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:35:28.738600 26750 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:35:28.739763 26750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:35:28.740094 26750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:35:28.741262 26750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:35:28.742071 26611 heartbeater.cc:503] Master 127.27.196.126:36809 was elected leader, sending a full tablet report...
I20240502 21:35:28.743204 26750 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:35:28.743341 26750 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:28.743482 26750 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:35:28.743567 26750 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:28.783987 26750 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.68:35953
I20240502 21:35:28.784085 26877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.68:35953 every 8 connection(s)
I20240502 21:35:28.785490 26750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:35:28.793711 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 26750
I20240502 21:35:28.794009 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 21:35:28.798938 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.69:0
--local_ip_for_outbound_sockets=127.27.196.69
--webserver_interface=127.27.196.69
--webserver_port=0
--tserver_master_addrs=127.27.196.126:36809
--builtin_ntp_servers=127.27.196.84:36263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:35:28.803303 26878 heartbeater.cc:347] Connected to a master server at 127.27.196.126:36809
I20240502 21:35:28.803786 26878 heartbeater.cc:464] Registering TS with master...
I20240502 21:35:28.805457 26878 heartbeater.cc:511] Master 127.27.196.126:36809 requested a full tablet report, sending...
I20240502 21:35:28.880910 26288 ts_manager.cc:194] Registered new tserver with Master: 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68:35953)
I20240502 21:35:28.882818 26288 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.68:45929
W20240502 21:35:29.226418 26882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:35:29.226758 26882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:35:29.226898 26882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:35:29.234922 26882 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:35:29.235087 26882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:35:29.235288 26882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.69
I20240502 21:35:29.244985 26882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:36263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.27.196.69
--webserver_port=0
--tserver_master_addrs=127.27.196.126:36809
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.69
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:35:29.246479 26882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:35:29.248386 26882 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:35:29.262187 26888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:29.262302 26889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:29.262830 26891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:35:29.263408 26882 server_base.cc:1034] running on GCE node
I20240502 21:35:29.264091 26882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:35:29.265126 26882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:35:29.266376 26882 hybrid_clock.cc:648] HybridClock initialized: now 1714685729266294 us; error 68 us; skew 500 ppm
I20240502 21:35:29.266764 26882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:35:29.269418 26882 webserver.cc:460] Webserver started at http://127.27.196.69:37245/ using document root <none> and password file <none>
I20240502 21:35:29.270304 26882 fs_manager.cc:359] Metadata directory not provided
I20240502 21:35:29.270468 26882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:35:29.270846 26882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:35:29.274022 26882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "c609c27f61b9436d95dc53deaa21a3b0"
format_stamp: "Formatted at 2024-05-02 21:35:29 on dist-test-slave-3bs1"
I20240502 21:35:29.275007 26882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "c609c27f61b9436d95dc53deaa21a3b0"
format_stamp: "Formatted at 2024-05-02 21:35:29 on dist-test-slave-3bs1"
I20240502 21:35:29.281486 26882 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.000s
I20240502 21:35:29.285465 26897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:29.286963 26882 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240502 21:35:29.287186 26882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c609c27f61b9436d95dc53deaa21a3b0"
format_stamp: "Formatted at 2024-05-02 21:35:29 on dist-test-slave-3bs1"
I20240502 21:35:29.287492 26882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:35:29.292913 26745 heartbeater.cc:503] Master 127.27.196.126:36809 was elected leader, sending a full tablet report...
I20240502 21:35:29.330266 26882 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:35:29.331403 26882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:35:29.331732 26882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:35:29.332880 26882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:35:29.334651 26882 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:35:29.334761 26882 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:29.334897 26882 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:35:29.334975 26882 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:29.375941 26882 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.69:43269
I20240502 21:35:29.375984 27011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.69:43269 every 8 connection(s)
I20240502 21:35:29.377895 26882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:35:29.386829 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 26882
I20240502 21:35:29.387068 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 21:35:29.396214 27013 heartbeater.cc:347] Connected to a master server at 127.27.196.126:36809
I20240502 21:35:29.396600 27013 heartbeater.cc:464] Registering TS with master...
I20240502 21:35:29.397615 27013 heartbeater.cc:511] Master 127.27.196.126:36809 requested a full tablet report, sending...
I20240502 21:35:29.453064 26288 ts_manager.cc:194] Registered new tserver with Master: c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69:43269)
I20240502 21:35:29.454138 26288 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.69:48533
I20240502 21:35:29.465610 28433 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 21:35:29.490518 28433 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 21:35:29.490687 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 26882
I20240502 21:35:29.503537 27007 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:35:29.592718 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 26882
I20240502 21:35:29.616292 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 26750
I20240502 21:35:29.627444 26873 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:35:29.714277 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 26750
I20240502 21:35:33.489832 26288 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55362:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240502 21:35:33.497128 26288 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 21:35:33.536868 26409 tablet_service.cc:1467] Processing CreateTablet for tablet a3a929c153c348ea8a0b4e7c2cf838f6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7277b1f7f8b747f4a8e65521bf4af6c0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:33.538703 26409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a929c153c348ea8a0b4e7c2cf838f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:33.537037 26410 tablet_service.cc:1467] Processing CreateTablet for tablet 8747e6794c084510a5b9e1f34c15c733 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7277b1f7f8b747f4a8e65521bf4af6c0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:33.539444 26410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8747e6794c084510a5b9e1f34c15c733. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:33.536823 26408 tablet_service.cc:1467] Processing CreateTablet for tablet c06cdd9252864256a6870171f0f4c09c (DEFAULT_TABLE table=rebalance_test_table_0 [id=7277b1f7f8b747f4a8e65521bf4af6c0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:33.539840 26678 tablet_service.cc:1467] Processing CreateTablet for tablet 8747e6794c084510a5b9e1f34c15c733 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7277b1f7f8b747f4a8e65521bf4af6c0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:33.540702 26408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c06cdd9252864256a6870171f0f4c09c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:33.541299 26678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8747e6794c084510a5b9e1f34c15c733. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:33.542011 26676 tablet_service.cc:1467] Processing CreateTablet for tablet c06cdd9252864256a6870171f0f4c09c (DEFAULT_TABLE table=rebalance_test_table_0 [id=7277b1f7f8b747f4a8e65521bf4af6c0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:33.542537 26676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c06cdd9252864256a6870171f0f4c09c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:33.543213 26677 tablet_service.cc:1467] Processing CreateTablet for tablet a3a929c153c348ea8a0b4e7c2cf838f6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7277b1f7f8b747f4a8e65521bf4af6c0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:33.543663 26677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a929c153c348ea8a0b4e7c2cf838f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:33.549110 26542 tablet_service.cc:1467] Processing CreateTablet for tablet c06cdd9252864256a6870171f0f4c09c (DEFAULT_TABLE table=rebalance_test_table_0 [id=7277b1f7f8b747f4a8e65521bf4af6c0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:33.549153 26543 tablet_service.cc:1467] Processing CreateTablet for tablet a3a929c153c348ea8a0b4e7c2cf838f6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7277b1f7f8b747f4a8e65521bf4af6c0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:33.550226 26544 tablet_service.cc:1467] Processing CreateTablet for tablet 8747e6794c084510a5b9e1f34c15c733 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7277b1f7f8b747f4a8e65521bf4af6c0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:33.550949 26542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c06cdd9252864256a6870171f0f4c09c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:33.551491 26544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8747e6794c084510a5b9e1f34c15c733. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:33.551918 26543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a929c153c348ea8a0b4e7c2cf838f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:33.566816 27038 tablet_bootstrap.cc:492] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc: Bootstrap starting.
I20240502 21:35:33.571656 27038 tablet_bootstrap.cc:654] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:33.574993 27038 log.cc:824] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc: Log is configured to *not* fsync() on all Append() calls
I20240502 21:35:33.578209 27039 tablet_bootstrap.cc:492] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0: Bootstrap starting.
I20240502 21:35:33.581451 27039 tablet_bootstrap.cc:654] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:33.582969 27039 log.cc:824] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0: Log is configured to *not* fsync() on all Append() calls
I20240502 21:35:33.583319 27041 tablet_bootstrap.cc:492] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092: Bootstrap starting.
I20240502 21:35:33.587653 27041 tablet_bootstrap.cc:654] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:33.587791 27039 tablet_bootstrap.cc:492] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0: No bootstrap required, opened a new log
I20240502 21:35:33.588131 27039 ts_tablet_manager.cc:1367] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0: Time spent bootstrapping tablet: real 0.010s	user 0.008s	sys 0.000s
I20240502 21:35:33.588362 27038 tablet_bootstrap.cc:492] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc: No bootstrap required, opened a new log
I20240502 21:35:33.588651 27038 ts_tablet_manager.cc:1367] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc: Time spent bootstrapping tablet: real 0.022s	user 0.009s	sys 0.008s
I20240502 21:35:33.589179 27041 log.cc:824] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092: Log is configured to *not* fsync() on all Append() calls
I20240502 21:35:33.592665 27041 tablet_bootstrap.cc:492] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092: No bootstrap required, opened a new log
I20240502 21:35:33.593017 27041 ts_tablet_manager.cc:1367] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092: Time spent bootstrapping tablet: real 0.010s	user 0.008s	sys 0.000s
I20240502 21:35:33.595902 27038 raft_consensus.cc:348] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:33.596329 27038 raft_consensus.cc:374] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:33.596460 27038 raft_consensus.cc:729] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63266faf827046629e39afa765b144bc, State: Initialized, Role: FOLLOWER
I20240502 21:35:33.597183 27038 consensus_queue.cc:260] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:33.597915 27039 raft_consensus.cc:348] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:33.598409 27039 raft_consensus.cc:374] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:33.598533 27039 raft_consensus.cc:729] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb27977c77564d38afe6e64c816eacd0, State: Initialized, Role: FOLLOWER
I20240502 21:35:33.599280 27039 consensus_queue.cc:260] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:33.600102 27038 ts_tablet_manager.cc:1398] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc: Time spent starting tablet: real 0.011s	user 0.011s	sys 0.000s
I20240502 21:35:33.600661 27038 tablet_bootstrap.cc:492] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc: Bootstrap starting.
I20240502 21:35:33.602607 27038 tablet_bootstrap.cc:654] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:33.605232 27041 raft_consensus.cc:348] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:33.605937 27041 raft_consensus.cc:374] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:33.606065 27041 raft_consensus.cc:729] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b1043ab39c04568814296d029600092, State: Initialized, Role: FOLLOWER
I20240502 21:35:33.607758 27041 consensus_queue.cc:260] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:33.610347 27041 ts_tablet_manager.cc:1398] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092: Time spent starting tablet: real 0.017s	user 0.016s	sys 0.000s
I20240502 21:35:33.610344 27039 ts_tablet_manager.cc:1398] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0: Time spent starting tablet: real 0.022s	user 0.004s	sys 0.017s
I20240502 21:35:33.610853 27039 tablet_bootstrap.cc:492] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0: Bootstrap starting.
I20240502 21:35:33.612882 27039 tablet_bootstrap.cc:654] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:33.614436 27041 tablet_bootstrap.cc:492] T c06cdd9252864256a6870171f0f4c09c P 5b1043ab39c04568814296d029600092: Bootstrap starting.
I20240502 21:35:33.615711 27038 tablet_bootstrap.cc:492] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc: No bootstrap required, opened a new log
I20240502 21:35:33.615923 27038 ts_tablet_manager.cc:1367] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc: Time spent bootstrapping tablet: real 0.015s	user 0.006s	sys 0.005s
I20240502 21:35:33.617520 27041 tablet_bootstrap.cc:654] T c06cdd9252864256a6870171f0f4c09c P 5b1043ab39c04568814296d029600092: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:33.617390 27038 raft_consensus.cc:348] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:33.617686 27038 raft_consensus.cc:374] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:33.617805 27038 raft_consensus.cc:729] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63266faf827046629e39afa765b144bc, State: Initialized, Role: FOLLOWER
I20240502 21:35:33.618091 27038 consensus_queue.cc:260] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:33.618727 27038 ts_tablet_manager.cc:1398] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 21:35:33.619040 27038 tablet_bootstrap.cc:492] T c06cdd9252864256a6870171f0f4c09c P 63266faf827046629e39afa765b144bc: Bootstrap starting.
I20240502 21:35:33.619913 27039 tablet_bootstrap.cc:492] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0: No bootstrap required, opened a new log
I20240502 21:35:33.620224 27039 ts_tablet_manager.cc:1367] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.004s
I20240502 21:35:33.620857 27038 tablet_bootstrap.cc:654] T c06cdd9252864256a6870171f0f4c09c P 63266faf827046629e39afa765b144bc: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:33.621953 27039 raft_consensus.cc:348] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:33.622355 27039 raft_consensus.cc:374] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:33.622529 27039 raft_consensus.cc:729] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb27977c77564d38afe6e64c816eacd0, State: Initialized, Role: FOLLOWER
I20240502 21:35:33.622898 27039 consensus_queue.cc:260] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:33.623752 27039 ts_tablet_manager.cc:1398] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 21:35:33.624200 27039 tablet_bootstrap.cc:492] T a3a929c153c348ea8a0b4e7c2cf838f6 P cb27977c77564d38afe6e64c816eacd0: Bootstrap starting.
I20240502 21:35:33.626942 27039 tablet_bootstrap.cc:654] T a3a929c153c348ea8a0b4e7c2cf838f6 P cb27977c77564d38afe6e64c816eacd0: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:33.629412 27038 tablet_bootstrap.cc:492] T c06cdd9252864256a6870171f0f4c09c P 63266faf827046629e39afa765b144bc: No bootstrap required, opened a new log
I20240502 21:35:33.629634 27038 ts_tablet_manager.cc:1367] T c06cdd9252864256a6870171f0f4c09c P 63266faf827046629e39afa765b144bc: Time spent bootstrapping tablet: real 0.011s	user 0.002s	sys 0.007s
I20240502 21:35:33.629802 27041 tablet_bootstrap.cc:492] T c06cdd9252864256a6870171f0f4c09c P 5b1043ab39c04568814296d029600092: No bootstrap required, opened a new log
I20240502 21:35:33.630035 27041 ts_tablet_manager.cc:1367] T c06cdd9252864256a6870171f0f4c09c P 5b1043ab39c04568814296d029600092: Time spent bootstrapping tablet: real 0.019s	user 0.007s	sys 0.007s
I20240502 21:35:33.630504 27038 raft_consensus.cc:348] T c06cdd9252864256a6870171f0f4c09c P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:33.630776 27038 raft_consensus.cc:374] T c06cdd9252864256a6870171f0f4c09c P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:33.630865 27038 raft_consensus.cc:729] T c06cdd9252864256a6870171f0f4c09c P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63266faf827046629e39afa765b144bc, State: Initialized, Role: FOLLOWER
I20240502 21:35:33.631181 27038 consensus_queue.cc:260] T c06cdd9252864256a6870171f0f4c09c P 63266faf827046629e39afa765b144bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:33.631302 27041 raft_consensus.cc:348] T c06cdd9252864256a6870171f0f4c09c P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:33.631515 27041 raft_consensus.cc:374] T c06cdd9252864256a6870171f0f4c09c P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:33.631601 27041 raft_consensus.cc:729] T c06cdd9252864256a6870171f0f4c09c P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b1043ab39c04568814296d029600092, State: Initialized, Role: FOLLOWER
I20240502 21:35:33.631949 27038 ts_tablet_manager.cc:1398] T c06cdd9252864256a6870171f0f4c09c P 63266faf827046629e39afa765b144bc: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.003s
I20240502 21:35:33.631852 27041 consensus_queue.cc:260] T c06cdd9252864256a6870171f0f4c09c P 5b1043ab39c04568814296d029600092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:33.632359 27041 ts_tablet_manager.cc:1398] T c06cdd9252864256a6870171f0f4c09c P 5b1043ab39c04568814296d029600092: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 21:35:33.632668 27041 tablet_bootstrap.cc:492] T a3a929c153c348ea8a0b4e7c2cf838f6 P 5b1043ab39c04568814296d029600092: Bootstrap starting.
I20240502 21:35:33.634223 27041 tablet_bootstrap.cc:654] T a3a929c153c348ea8a0b4e7c2cf838f6 P 5b1043ab39c04568814296d029600092: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:33.634420 27039 tablet_bootstrap.cc:492] T a3a929c153c348ea8a0b4e7c2cf838f6 P cb27977c77564d38afe6e64c816eacd0: No bootstrap required, opened a new log
I20240502 21:35:33.634652 27039 ts_tablet_manager.cc:1367] T a3a929c153c348ea8a0b4e7c2cf838f6 P cb27977c77564d38afe6e64c816eacd0: Time spent bootstrapping tablet: real 0.011s	user 0.004s	sys 0.003s
I20240502 21:35:33.635594 27039 raft_consensus.cc:348] T a3a929c153c348ea8a0b4e7c2cf838f6 P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:33.635890 27039 raft_consensus.cc:374] T a3a929c153c348ea8a0b4e7c2cf838f6 P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:33.636014 27039 raft_consensus.cc:729] T a3a929c153c348ea8a0b4e7c2cf838f6 P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb27977c77564d38afe6e64c816eacd0, State: Initialized, Role: FOLLOWER
I20240502 21:35:33.636319 27039 consensus_queue.cc:260] T a3a929c153c348ea8a0b4e7c2cf838f6 P cb27977c77564d38afe6e64c816eacd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:33.637046 27039 ts_tablet_manager.cc:1398] T a3a929c153c348ea8a0b4e7c2cf838f6 P cb27977c77564d38afe6e64c816eacd0: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 21:35:33.639905 27041 tablet_bootstrap.cc:492] T a3a929c153c348ea8a0b4e7c2cf838f6 P 5b1043ab39c04568814296d029600092: No bootstrap required, opened a new log
I20240502 21:35:33.640254 27041 ts_tablet_manager.cc:1367] T a3a929c153c348ea8a0b4e7c2cf838f6 P 5b1043ab39c04568814296d029600092: Time spent bootstrapping tablet: real 0.008s	user 0.002s	sys 0.005s
I20240502 21:35:33.641254 27041 raft_consensus.cc:348] T a3a929c153c348ea8a0b4e7c2cf838f6 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:33.641506 27041 raft_consensus.cc:374] T a3a929c153c348ea8a0b4e7c2cf838f6 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:33.641599 27041 raft_consensus.cc:729] T a3a929c153c348ea8a0b4e7c2cf838f6 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b1043ab39c04568814296d029600092, State: Initialized, Role: FOLLOWER
I20240502 21:35:33.641892 27041 consensus_queue.cc:260] T a3a929c153c348ea8a0b4e7c2cf838f6 P 5b1043ab39c04568814296d029600092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:33.642498 27041 ts_tablet_manager.cc:1398] T a3a929c153c348ea8a0b4e7c2cf838f6 P 5b1043ab39c04568814296d029600092: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
W20240502 21:35:33.648751 26612 tablet.cc:2369] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:35:33.653215 27044 raft_consensus.cc:482] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:35:33.653569 27044 raft_consensus.cc:504] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:33.655849 27044 leader_election.cc:288] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb27977c77564d38afe6e64c816eacd0 (127.27.196.65:36209), 5b1043ab39c04568814296d029600092 (127.27.196.66:41285)
I20240502 21:35:33.666461 26565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a929c153c348ea8a0b4e7c2cf838f6" candidate_uuid: "63266faf827046629e39afa765b144bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1043ab39c04568814296d029600092" is_pre_election: true
I20240502 21:35:33.666461 26430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a929c153c348ea8a0b4e7c2cf838f6" candidate_uuid: "63266faf827046629e39afa765b144bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb27977c77564d38afe6e64c816eacd0" is_pre_election: true
I20240502 21:35:33.667205 26430 raft_consensus.cc:2454] T a3a929c153c348ea8a0b4e7c2cf838f6 P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63266faf827046629e39afa765b144bc in term 0.
I20240502 21:35:33.667204 26565 raft_consensus.cc:2454] T a3a929c153c348ea8a0b4e7c2cf838f6 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63266faf827046629e39afa765b144bc in term 0.
I20240502 21:35:33.668041 26632 leader_election.cc:302] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63266faf827046629e39afa765b144bc, cb27977c77564d38afe6e64c816eacd0; no voters: 
I20240502 21:35:33.668526 27044 raft_consensus.cc:2781] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:35:33.668720 27044 raft_consensus.cc:482] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:35:33.668828 27044 raft_consensus.cc:3037] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:33.671450 27044 raft_consensus.cc:504] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:33.672195 27044 leader_election.cc:288] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [CANDIDATE]: Term 1 election: Requested vote from peers cb27977c77564d38afe6e64c816eacd0 (127.27.196.65:36209), 5b1043ab39c04568814296d029600092 (127.27.196.66:41285)
I20240502 21:35:33.672916 26430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a929c153c348ea8a0b4e7c2cf838f6" candidate_uuid: "63266faf827046629e39afa765b144bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb27977c77564d38afe6e64c816eacd0"
I20240502 21:35:33.673040 26565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a929c153c348ea8a0b4e7c2cf838f6" candidate_uuid: "63266faf827046629e39afa765b144bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1043ab39c04568814296d029600092"
I20240502 21:35:33.673144 26430 raft_consensus.cc:3037] T a3a929c153c348ea8a0b4e7c2cf838f6 P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:33.673296 26565 raft_consensus.cc:3037] T a3a929c153c348ea8a0b4e7c2cf838f6 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:33.675705 26565 raft_consensus.cc:2454] T a3a929c153c348ea8a0b4e7c2cf838f6 P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63266faf827046629e39afa765b144bc in term 1.
I20240502 21:35:33.675705 26430 raft_consensus.cc:2454] T a3a929c153c348ea8a0b4e7c2cf838f6 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63266faf827046629e39afa765b144bc in term 1.
I20240502 21:35:33.676405 26632 leader_election.cc:302] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b1043ab39c04568814296d029600092, 63266faf827046629e39afa765b144bc; no voters: 
I20240502 21:35:33.677071 27044 raft_consensus.cc:2781] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:33.677740 27044 raft_consensus.cc:686] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [term 1 LEADER]: Becoming Leader. State: Replica: 63266faf827046629e39afa765b144bc, State: Running, Role: LEADER
I20240502 21:35:33.678607 27044 consensus_queue.cc:237] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:33.685093 26288 catalog_manager.cc:5484] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 63266faf827046629e39afa765b144bc (127.27.196.67). New cstate: current_term: 1 leader_uuid: "63266faf827046629e39afa765b144bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } health_report { overall_health: HEALTHY } } }
I20240502 21:35:33.695395 27045 raft_consensus.cc:482] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:35:33.695875 27045 raft_consensus.cc:504] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:33.697986 27045 leader_election.cc:288] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b1043ab39c04568814296d029600092 (127.27.196.66:41285), 63266faf827046629e39afa765b144bc (127.27.196.67:39203)
W20240502 21:35:33.701803 26746 tablet.cc:2369] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:35:33.708010 26565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c06cdd9252864256a6870171f0f4c09c" candidate_uuid: "cb27977c77564d38afe6e64c816eacd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1043ab39c04568814296d029600092" is_pre_election: true
I20240502 21:35:33.708303 26565 raft_consensus.cc:2454] T c06cdd9252864256a6870171f0f4c09c P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb27977c77564d38afe6e64c816eacd0 in term 0.
I20240502 21:35:33.708235 26698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c06cdd9252864256a6870171f0f4c09c" candidate_uuid: "cb27977c77564d38afe6e64c816eacd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63266faf827046629e39afa765b144bc" is_pre_election: true
I20240502 21:35:33.708722 26698 raft_consensus.cc:2454] T c06cdd9252864256a6870171f0f4c09c P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb27977c77564d38afe6e64c816eacd0 in term 0.
I20240502 21:35:33.708899 26363 leader_election.cc:302] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b1043ab39c04568814296d029600092, cb27977c77564d38afe6e64c816eacd0; no voters: 
I20240502 21:35:33.709448 27045 raft_consensus.cc:2781] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:35:33.709582 27045 raft_consensus.cc:482] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:35:33.709683 27045 raft_consensus.cc:3037] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:33.712081 27045 raft_consensus.cc:504] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:33.712774 27045 leader_election.cc:288] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [CANDIDATE]: Term 1 election: Requested vote from peers 5b1043ab39c04568814296d029600092 (127.27.196.66:41285), 63266faf827046629e39afa765b144bc (127.27.196.67:39203)
I20240502 21:35:33.713387 26565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c06cdd9252864256a6870171f0f4c09c" candidate_uuid: "cb27977c77564d38afe6e64c816eacd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1043ab39c04568814296d029600092"
I20240502 21:35:33.713402 26698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c06cdd9252864256a6870171f0f4c09c" candidate_uuid: "cb27977c77564d38afe6e64c816eacd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63266faf827046629e39afa765b144bc"
I20240502 21:35:33.713616 26698 raft_consensus.cc:3037] T c06cdd9252864256a6870171f0f4c09c P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:33.713631 26565 raft_consensus.cc:3037] T c06cdd9252864256a6870171f0f4c09c P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:33.715751 26565 raft_consensus.cc:2454] T c06cdd9252864256a6870171f0f4c09c P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb27977c77564d38afe6e64c816eacd0 in term 1.
I20240502 21:35:33.715934 26698 raft_consensus.cc:2454] T c06cdd9252864256a6870171f0f4c09c P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb27977c77564d38afe6e64c816eacd0 in term 1.
I20240502 21:35:33.716301 26363 leader_election.cc:302] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b1043ab39c04568814296d029600092, cb27977c77564d38afe6e64c816eacd0; no voters: 
I20240502 21:35:33.716650 27045 raft_consensus.cc:2781] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:33.717141 27045 raft_consensus.cc:686] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [term 1 LEADER]: Becoming Leader. State: Replica: cb27977c77564d38afe6e64c816eacd0, State: Running, Role: LEADER
I20240502 21:35:33.717758 27045 consensus_queue.cc:237] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:33.723228 26287 catalog_manager.cc:5484] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb27977c77564d38afe6e64c816eacd0 (127.27.196.65). New cstate: current_term: 1 leader_uuid: "cb27977c77564d38afe6e64c816eacd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } health_report { overall_health: UNKNOWN } } }
W20240502 21:35:33.817873 26477 tablet.cc:2369] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:35:33.884946 27047 raft_consensus.cc:482] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:35:33.885178 27047 raft_consensus.cc:504] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:33.887331 27047 leader_election.cc:288] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb27977c77564d38afe6e64c816eacd0 (127.27.196.65:36209), 63266faf827046629e39afa765b144bc (127.27.196.67:39203)
I20240502 21:35:33.898471 26698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8747e6794c084510a5b9e1f34c15c733" candidate_uuid: "5b1043ab39c04568814296d029600092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63266faf827046629e39afa765b144bc" is_pre_election: true
I20240502 21:35:33.898555 26430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8747e6794c084510a5b9e1f34c15c733" candidate_uuid: "5b1043ab39c04568814296d029600092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb27977c77564d38afe6e64c816eacd0" is_pre_election: true
I20240502 21:35:33.898790 26698 raft_consensus.cc:2454] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b1043ab39c04568814296d029600092 in term 0.
I20240502 21:35:33.898813 26430 raft_consensus.cc:2454] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b1043ab39c04568814296d029600092 in term 0.
I20240502 21:35:33.899423 26498 leader_election.cc:302] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b1043ab39c04568814296d029600092, cb27977c77564d38afe6e64c816eacd0; no voters: 
I20240502 21:35:33.899924 27047 raft_consensus.cc:2781] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:35:33.900064 27047 raft_consensus.cc:482] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:35:33.900171 27047 raft_consensus.cc:3037] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:33.902673 27047 raft_consensus.cc:504] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:33.903610 27047 leader_election.cc:288] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [CANDIDATE]: Term 1 election: Requested vote from peers cb27977c77564d38afe6e64c816eacd0 (127.27.196.65:36209), 63266faf827046629e39afa765b144bc (127.27.196.67:39203)
I20240502 21:35:33.904232 26430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8747e6794c084510a5b9e1f34c15c733" candidate_uuid: "5b1043ab39c04568814296d029600092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb27977c77564d38afe6e64c816eacd0"
I20240502 21:35:33.904314 26698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8747e6794c084510a5b9e1f34c15c733" candidate_uuid: "5b1043ab39c04568814296d029600092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63266faf827046629e39afa765b144bc"
I20240502 21:35:33.904515 26430 raft_consensus.cc:3037] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:33.904572 26698 raft_consensus.cc:3037] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:33.906643 26698 raft_consensus.cc:2454] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b1043ab39c04568814296d029600092 in term 1.
I20240502 21:35:33.906646 26430 raft_consensus.cc:2454] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b1043ab39c04568814296d029600092 in term 1.
I20240502 21:35:33.907225 26498 leader_election.cc:302] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b1043ab39c04568814296d029600092, cb27977c77564d38afe6e64c816eacd0; no voters: 
I20240502 21:35:33.907599 27047 raft_consensus.cc:2781] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:33.908102 27047 raft_consensus.cc:686] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [term 1 LEADER]: Becoming Leader. State: Replica: 5b1043ab39c04568814296d029600092, State: Running, Role: LEADER
I20240502 21:35:33.908715 27047 consensus_queue.cc:237] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:33.914220 26287 catalog_manager.cc:5484] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b1043ab39c04568814296d029600092 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "5b1043ab39c04568814296d029600092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } health_report { overall_health: UNKNOWN } } }
I20240502 21:35:34.093669 27044 consensus_queue.cc:1035] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:35:34.110857 27055 consensus_queue.cc:1035] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:35:34.334467 27045 consensus_queue.cc:1035] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:35:34.335402 27045 consensus_queue.cc:1035] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:35:34.406832 27047 consensus_queue.cc:1035] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:35:34.413780 27047 consensus_queue.cc:1035] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:35:34.440626 27065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:35:34.441013 27065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:35:34.448908 27065 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
W20240502 21:35:34.747241 26363 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.196.66:41285, user_credentials={real_user=slave}} blocked reactor thread for 55051.4us

INSERT report
    rows total: 2000
    time total: 293.835 ms
  time per row: 0.146918 ms
I20240502 21:35:34.926038 26283 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55362:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240502 21:35:34.927080 26283 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 21:35:34.939550 26677 tablet_service.cc:1467] Processing CreateTablet for tablet 1331eade23b14aee86ceeb067c93a9d0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=cc5055b898c74d0ea2d82c8d8f1f5bb4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:34.939635 26410 tablet_service.cc:1467] Processing CreateTablet for tablet 1331eade23b14aee86ceeb067c93a9d0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=cc5055b898c74d0ea2d82c8d8f1f5bb4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:34.939635 26543 tablet_service.cc:1467] Processing CreateTablet for tablet 140f7280768e4ad592309e9b7d6a26da (DEFAULT_TABLE table=rebalance_test_table_1 [id=cc5055b898c74d0ea2d82c8d8f1f5bb4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:34.940251 26543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 140f7280768e4ad592309e9b7d6a26da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:34.939571 26544 tablet_service.cc:1467] Processing CreateTablet for tablet 1331eade23b14aee86ceeb067c93a9d0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=cc5055b898c74d0ea2d82c8d8f1f5bb4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:34.941097 26677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1331eade23b14aee86ceeb067c93a9d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:34.941341 26544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1331eade23b14aee86ceeb067c93a9d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:34.940486 26410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1331eade23b14aee86ceeb067c93a9d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:34.941527 26408 tablet_service.cc:1467] Processing CreateTablet for tablet 140f7280768e4ad592309e9b7d6a26da (DEFAULT_TABLE table=rebalance_test_table_1 [id=cc5055b898c74d0ea2d82c8d8f1f5bb4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:34.942195 26408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 140f7280768e4ad592309e9b7d6a26da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:34.945544 26676 tablet_service.cc:1467] Processing CreateTablet for tablet 140f7280768e4ad592309e9b7d6a26da (DEFAULT_TABLE table=rebalance_test_table_1 [id=cc5055b898c74d0ea2d82c8d8f1f5bb4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:34.946118 26676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 140f7280768e4ad592309e9b7d6a26da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:34.950692 27135 tablet_bootstrap.cc:492] T 1331eade23b14aee86ceeb067c93a9d0 P 5b1043ab39c04568814296d029600092: Bootstrap starting.
I20240502 21:35:34.951668 26410 tablet_service.cc:1467] Processing CreateTablet for tablet 35508ded8421457288bdef00bf12f698 (DEFAULT_TABLE table=rebalance_test_table_1 [id=cc5055b898c74d0ea2d82c8d8f1f5bb4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:34.951676 26544 tablet_service.cc:1467] Processing CreateTablet for tablet 35508ded8421457288bdef00bf12f698 (DEFAULT_TABLE table=rebalance_test_table_1 [id=cc5055b898c74d0ea2d82c8d8f1f5bb4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:34.952210 26410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35508ded8421457288bdef00bf12f698. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:34.952407 26544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35508ded8421457288bdef00bf12f698. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:34.953127 26678 tablet_service.cc:1467] Processing CreateTablet for tablet 35508ded8421457288bdef00bf12f698 (DEFAULT_TABLE table=rebalance_test_table_1 [id=cc5055b898c74d0ea2d82c8d8f1f5bb4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:34.954978 26678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35508ded8421457288bdef00bf12f698. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:34.959350 27136 tablet_bootstrap.cc:492] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0: Bootstrap starting.
I20240502 21:35:34.961117 27136 tablet_bootstrap.cc:654] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:34.970058 27135 tablet_bootstrap.cc:654] T 1331eade23b14aee86ceeb067c93a9d0 P 5b1043ab39c04568814296d029600092: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:34.972857 27138 tablet_bootstrap.cc:492] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc: Bootstrap starting.
I20240502 21:35:34.975024 27138 tablet_bootstrap.cc:654] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:34.979864 27135 tablet_bootstrap.cc:492] T 1331eade23b14aee86ceeb067c93a9d0 P 5b1043ab39c04568814296d029600092: No bootstrap required, opened a new log
I20240502 21:35:34.979903 27136 tablet_bootstrap.cc:492] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0: No bootstrap required, opened a new log
I20240502 21:35:34.980074 27135 ts_tablet_manager.cc:1367] T 1331eade23b14aee86ceeb067c93a9d0 P 5b1043ab39c04568814296d029600092: Time spent bootstrapping tablet: real 0.030s	user 0.009s	sys 0.000s
I20240502 21:35:34.980101 27136 ts_tablet_manager.cc:1367] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0: Time spent bootstrapping tablet: real 0.021s	user 0.010s	sys 0.007s
I20240502 21:35:34.980150 27138 tablet_bootstrap.cc:492] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc: No bootstrap required, opened a new log
I20240502 21:35:34.980402 27138 ts_tablet_manager.cc:1367] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc: Time spent bootstrapping tablet: real 0.008s	user 0.005s	sys 0.000s
I20240502 21:35:34.981086 27135 raft_consensus.cc:348] T 1331eade23b14aee86ceeb067c93a9d0 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:34.981328 27135 raft_consensus.cc:374] T 1331eade23b14aee86ceeb067c93a9d0 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:34.981443 27135 raft_consensus.cc:729] T 1331eade23b14aee86ceeb067c93a9d0 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b1043ab39c04568814296d029600092, State: Initialized, Role: FOLLOWER
I20240502 21:35:34.981104 27136 raft_consensus.cc:348] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:34.981547 27136 raft_consensus.cc:374] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:34.981654 27136 raft_consensus.cc:729] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb27977c77564d38afe6e64c816eacd0, State: Initialized, Role: FOLLOWER
I20240502 21:35:34.981591 27138 raft_consensus.cc:348] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:34.981719 27135 consensus_queue.cc:260] T 1331eade23b14aee86ceeb067c93a9d0 P 5b1043ab39c04568814296d029600092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:34.981931 27138 raft_consensus.cc:374] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:34.981928 27136 consensus_queue.cc:260] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:34.982096 27138 raft_consensus.cc:729] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63266faf827046629e39afa765b144bc, State: Initialized, Role: FOLLOWER
I20240502 21:35:34.982523 27138 consensus_queue.cc:260] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:34.983609 27138 ts_tablet_manager.cc:1398] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 21:35:34.984138 27138 tablet_bootstrap.cc:492] T 1331eade23b14aee86ceeb067c93a9d0 P 63266faf827046629e39afa765b144bc: Bootstrap starting.
I20240502 21:35:34.984210 27136 ts_tablet_manager.cc:1398] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.003s
I20240502 21:35:34.984531 27136 tablet_bootstrap.cc:492] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0: Bootstrap starting.
I20240502 21:35:34.986218 27115 raft_consensus.cc:482] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:35:34.986438 27136 tablet_bootstrap.cc:654] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:34.986430 27115 raft_consensus.cc:504] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:34.986761 27138 tablet_bootstrap.cc:654] T 1331eade23b14aee86ceeb067c93a9d0 P 63266faf827046629e39afa765b144bc: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:34.987515 27115 leader_election.cc:288] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b1043ab39c04568814296d029600092 (127.27.196.66:41285), 63266faf827046629e39afa765b144bc (127.27.196.67:39203)
I20240502 21:35:34.988535 26565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1331eade23b14aee86ceeb067c93a9d0" candidate_uuid: "cb27977c77564d38afe6e64c816eacd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1043ab39c04568814296d029600092" is_pre_election: true
I20240502 21:35:34.988847 26565 raft_consensus.cc:2454] T 1331eade23b14aee86ceeb067c93a9d0 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb27977c77564d38afe6e64c816eacd0 in term 0.
I20240502 21:35:34.989408 26363 leader_election.cc:302] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b1043ab39c04568814296d029600092, cb27977c77564d38afe6e64c816eacd0; no voters: 
I20240502 21:35:34.989977 27136 tablet_bootstrap.cc:492] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0: No bootstrap required, opened a new log
I20240502 21:35:34.989929 26698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1331eade23b14aee86ceeb067c93a9d0" candidate_uuid: "cb27977c77564d38afe6e64c816eacd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63266faf827046629e39afa765b144bc" is_pre_election: true
I20240502 21:35:34.990155 27136 ts_tablet_manager.cc:1367] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240502 21:35:34.991043 27136 raft_consensus.cc:348] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:34.991353 27136 raft_consensus.cc:374] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:34.991521 27136 raft_consensus.cc:729] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb27977c77564d38afe6e64c816eacd0, State: Initialized, Role: FOLLOWER
W20240502 21:35:34.991333 26364 leader_election.cc:341] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63266faf827046629e39afa765b144bc (127.27.196.67:39203): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:35:34.991729 27136 consensus_queue.cc:260] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:34.992060 27072 raft_consensus.cc:2781] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:35:34.992164 27072 raft_consensus.cc:482] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:35:34.992275 27072 raft_consensus.cc:3037] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:34.992769 27136 ts_tablet_manager.cc:1398] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 21:35:34.993139 27136 tablet_bootstrap.cc:492] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0: Bootstrap starting.
I20240502 21:35:34.994989 27136 tablet_bootstrap.cc:654] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:34.996153 27135 ts_tablet_manager.cc:1398] T 1331eade23b14aee86ceeb067c93a9d0 P 5b1043ab39c04568814296d029600092: Time spent starting tablet: real 0.016s	user 0.002s	sys 0.000s
I20240502 21:35:34.996742 27072 raft_consensus.cc:504] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:34.998165 26565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1331eade23b14aee86ceeb067c93a9d0" candidate_uuid: "cb27977c77564d38afe6e64c816eacd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1043ab39c04568814296d029600092"
I20240502 21:35:34.998517 26565 raft_consensus.cc:3037] T 1331eade23b14aee86ceeb067c93a9d0 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:35.002658 26565 raft_consensus.cc:2454] T 1331eade23b14aee86ceeb067c93a9d0 P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb27977c77564d38afe6e64c816eacd0 in term 1.
I20240502 21:35:35.003276 26363 leader_election.cc:302] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b1043ab39c04568814296d029600092, cb27977c77564d38afe6e64c816eacd0; no voters: 
I20240502 21:35:35.004829 27115 raft_consensus.cc:2781] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:35.005086 27115 raft_consensus.cc:686] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [term 1 LEADER]: Becoming Leader. State: Replica: cb27977c77564d38afe6e64c816eacd0, State: Running, Role: LEADER
I20240502 21:35:35.005506 27115 consensus_queue.cc:237] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:35.007551 27072 leader_election.cc:288] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [CANDIDATE]: Term 1 election: Requested vote from peers 5b1043ab39c04568814296d029600092 (127.27.196.66:41285), 63266faf827046629e39afa765b144bc (127.27.196.67:39203)
I20240502 21:35:35.008111 27135 tablet_bootstrap.cc:492] T 140f7280768e4ad592309e9b7d6a26da P 5b1043ab39c04568814296d029600092: Bootstrap starting.
I20240502 21:35:35.008545 26698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1331eade23b14aee86ceeb067c93a9d0" candidate_uuid: "cb27977c77564d38afe6e64c816eacd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63266faf827046629e39afa765b144bc"
W20240502 21:35:35.009397 26364 leader_election.cc:341] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 63266faf827046629e39afa765b144bc (127.27.196.67:39203): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:35:35.012310 27135 tablet_bootstrap.cc:654] T 140f7280768e4ad592309e9b7d6a26da P 5b1043ab39c04568814296d029600092: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:35.020018 27135 tablet_bootstrap.cc:492] T 140f7280768e4ad592309e9b7d6a26da P 5b1043ab39c04568814296d029600092: No bootstrap required, opened a new log
I20240502 21:35:35.020452 27135 ts_tablet_manager.cc:1367] T 140f7280768e4ad592309e9b7d6a26da P 5b1043ab39c04568814296d029600092: Time spent bootstrapping tablet: real 0.012s	user 0.008s	sys 0.000s
I20240502 21:35:35.021081 26281 catalog_manager.cc:5484] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb27977c77564d38afe6e64c816eacd0 (127.27.196.65). New cstate: current_term: 1 leader_uuid: "cb27977c77564d38afe6e64c816eacd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } health_report { overall_health: UNKNOWN } } }
I20240502 21:35:35.021906 27135 raft_consensus.cc:348] T 140f7280768e4ad592309e9b7d6a26da P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:35.022347 27135 raft_consensus.cc:374] T 140f7280768e4ad592309e9b7d6a26da P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:35.022495 27135 raft_consensus.cc:729] T 140f7280768e4ad592309e9b7d6a26da P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b1043ab39c04568814296d029600092, State: Initialized, Role: FOLLOWER
I20240502 21:35:35.022984 27135 consensus_queue.cc:260] T 140f7280768e4ad592309e9b7d6a26da P 5b1043ab39c04568814296d029600092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:35.024853 27135 ts_tablet_manager.cc:1398] T 140f7280768e4ad592309e9b7d6a26da P 5b1043ab39c04568814296d029600092: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 21:35:35.026940 27136 tablet_bootstrap.cc:492] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0: No bootstrap required, opened a new log
I20240502 21:35:35.027154 27136 ts_tablet_manager.cc:1367] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0: Time spent bootstrapping tablet: real 0.034s	user 0.003s	sys 0.026s
I20240502 21:35:35.027994 27136 raft_consensus.cc:348] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:35.028213 27136 raft_consensus.cc:374] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:35.028309 27136 raft_consensus.cc:729] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb27977c77564d38afe6e64c816eacd0, State: Initialized, Role: FOLLOWER
I20240502 21:35:35.028589 27136 consensus_queue.cc:260] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:35.029394 27135 tablet_bootstrap.cc:492] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092: Bootstrap starting.
I20240502 21:35:35.029804 27136 ts_tablet_manager.cc:1398] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 21:35:35.032781 27118 raft_consensus.cc:482] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:35:35.033036 27118 raft_consensus.cc:504] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:35.033813 27135 tablet_bootstrap.cc:654] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:35.034257 27118 leader_election.cc:288] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb27977c77564d38afe6e64c816eacd0 (127.27.196.65:36209), 5b1043ab39c04568814296d029600092 (127.27.196.66:41285)
I20240502 21:35:35.035071 26429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35508ded8421457288bdef00bf12f698" candidate_uuid: "63266faf827046629e39afa765b144bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb27977c77564d38afe6e64c816eacd0" is_pre_election: true
I20240502 21:35:35.035377 26429 raft_consensus.cc:2454] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63266faf827046629e39afa765b144bc in term 0.
I20240502 21:35:35.035516 26565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35508ded8421457288bdef00bf12f698" candidate_uuid: "63266faf827046629e39afa765b144bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1043ab39c04568814296d029600092" is_pre_election: true
I20240502 21:35:35.036063 26632 leader_election.cc:302] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63266faf827046629e39afa765b144bc, cb27977c77564d38afe6e64c816eacd0; no voters: 
I20240502 21:35:35.036437 27118 raft_consensus.cc:2781] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:35:35.036576 27118 raft_consensus.cc:482] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:35:35.036687 27118 raft_consensus.cc:3037] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:35:35.037384 26632 leader_election.cc:341] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b1043ab39c04568814296d029600092 (127.27.196.66:41285): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:35:35.038210 27135 tablet_bootstrap.cc:492] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092: No bootstrap required, opened a new log
I20240502 21:35:35.038384 27135 ts_tablet_manager.cc:1367] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092: Time spent bootstrapping tablet: real 0.009s	user 0.005s	sys 0.000s
I20240502 21:35:35.039289 27135 raft_consensus.cc:348] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:35.039556 27135 raft_consensus.cc:374] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:35.039674 27135 raft_consensus.cc:729] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b1043ab39c04568814296d029600092, State: Initialized, Role: FOLLOWER
I20240502 21:35:35.039608 27118 raft_consensus.cc:504] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:35.039988 27135 consensus_queue.cc:260] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:35.040745 27138 tablet_bootstrap.cc:492] T 1331eade23b14aee86ceeb067c93a9d0 P 63266faf827046629e39afa765b144bc: No bootstrap required, opened a new log
I20240502 21:35:35.041031 27138 ts_tablet_manager.cc:1367] T 1331eade23b14aee86ceeb067c93a9d0 P 63266faf827046629e39afa765b144bc: Time spent bootstrapping tablet: real 0.057s	user 0.017s	sys 0.033s
I20240502 21:35:35.041338 26429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35508ded8421457288bdef00bf12f698" candidate_uuid: "63266faf827046629e39afa765b144bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb27977c77564d38afe6e64c816eacd0"
I20240502 21:35:35.041471 27135 ts_tablet_manager.cc:1398] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 21:35:35.041522 26565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35508ded8421457288bdef00bf12f698" candidate_uuid: "63266faf827046629e39afa765b144bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1043ab39c04568814296d029600092"
I20240502 21:35:35.041640 26429 raft_consensus.cc:3037] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:35.041739 26565 raft_consensus.cc:3037] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:35.042326 27138 raft_consensus.cc:348] T 1331eade23b14aee86ceeb067c93a9d0 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:35.042793 27138 raft_consensus.cc:374] T 1331eade23b14aee86ceeb067c93a9d0 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:35.042973 27138 raft_consensus.cc:729] T 1331eade23b14aee86ceeb067c93a9d0 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63266faf827046629e39afa765b144bc, State: Initialized, Role: FOLLOWER
I20240502 21:35:35.043464 27138 consensus_queue.cc:260] T 1331eade23b14aee86ceeb067c93a9d0 P 63266faf827046629e39afa765b144bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:35.045192 26429 raft_consensus.cc:2454] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63266faf827046629e39afa765b144bc in term 1.
I20240502 21:35:35.045449 26565 raft_consensus.cc:2454] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63266faf827046629e39afa765b144bc in term 1.
I20240502 21:35:35.045687 27138 ts_tablet_manager.cc:1398] T 1331eade23b14aee86ceeb067c93a9d0 P 63266faf827046629e39afa765b144bc: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 21:35:35.045964 26632 leader_election.cc:302] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63266faf827046629e39afa765b144bc, cb27977c77564d38afe6e64c816eacd0; no voters: 
I20240502 21:35:35.046101 27138 tablet_bootstrap.cc:492] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc: Bootstrap starting.
I20240502 21:35:35.040607 27118 leader_election.cc:288] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [CANDIDATE]: Term 1 election: Requested vote from peers cb27977c77564d38afe6e64c816eacd0 (127.27.196.65:36209), 5b1043ab39c04568814296d029600092 (127.27.196.66:41285)
I20240502 21:35:35.046401 27055 raft_consensus.cc:2781] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:35.046921 27055 raft_consensus.cc:686] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 1 LEADER]: Becoming Leader. State: Replica: 63266faf827046629e39afa765b144bc, State: Running, Role: LEADER
I20240502 21:35:35.047277 27055 consensus_queue.cc:237] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:35.049243 27138 tablet_bootstrap.cc:654] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:35.051525 26283 catalog_manager.cc:5484] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 63266faf827046629e39afa765b144bc (127.27.196.67). New cstate: current_term: 1 leader_uuid: "63266faf827046629e39afa765b144bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } health_report { overall_health: HEALTHY } } }
I20240502 21:35:35.052722 27138 tablet_bootstrap.cc:492] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc: No bootstrap required, opened a new log
I20240502 21:35:35.052883 27138 ts_tablet_manager.cc:1367] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.000s
I20240502 21:35:35.053799 27138 raft_consensus.cc:348] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:35.054026 27138 raft_consensus.cc:374] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:35.054136 27138 raft_consensus.cc:729] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63266faf827046629e39afa765b144bc, State: Initialized, Role: FOLLOWER
I20240502 21:35:35.054414 27138 consensus_queue.cc:260] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:35.054986 27138 ts_tablet_manager.cc:1398] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 21:35:35.209321 27115 raft_consensus.cc:482] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:35:35.209566 27115 raft_consensus.cc:504] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:35.210325 27115 leader_election.cc:288] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b1043ab39c04568814296d029600092 (127.27.196.66:41285), 63266faf827046629e39afa765b144bc (127.27.196.67:39203)
I20240502 21:35:35.210933 26698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "140f7280768e4ad592309e9b7d6a26da" candidate_uuid: "cb27977c77564d38afe6e64c816eacd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63266faf827046629e39afa765b144bc" is_pre_election: true
I20240502 21:35:35.210948 26565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "140f7280768e4ad592309e9b7d6a26da" candidate_uuid: "cb27977c77564d38afe6e64c816eacd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1043ab39c04568814296d029600092" is_pre_election: true
I20240502 21:35:35.211247 26565 raft_consensus.cc:2454] T 140f7280768e4ad592309e9b7d6a26da P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb27977c77564d38afe6e64c816eacd0 in term 0.
I20240502 21:35:35.211241 26698 raft_consensus.cc:2454] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb27977c77564d38afe6e64c816eacd0 in term 0.
I20240502 21:35:35.211766 26364 leader_election.cc:302] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63266faf827046629e39afa765b144bc, cb27977c77564d38afe6e64c816eacd0; no voters: 
I20240502 21:35:35.212119 27115 raft_consensus.cc:2781] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:35:35.212255 27115 raft_consensus.cc:482] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:35:35.212352 27115 raft_consensus.cc:3037] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:35.214448 27115 raft_consensus.cc:504] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:35.215176 27115 leader_election.cc:288] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [CANDIDATE]: Term 1 election: Requested vote from peers 5b1043ab39c04568814296d029600092 (127.27.196.66:41285), 63266faf827046629e39afa765b144bc (127.27.196.67:39203)
I20240502 21:35:35.215736 26565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "140f7280768e4ad592309e9b7d6a26da" candidate_uuid: "cb27977c77564d38afe6e64c816eacd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1043ab39c04568814296d029600092"
I20240502 21:35:35.215762 26698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "140f7280768e4ad592309e9b7d6a26da" candidate_uuid: "cb27977c77564d38afe6e64c816eacd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63266faf827046629e39afa765b144bc"
I20240502 21:35:35.215978 26565 raft_consensus.cc:3037] T 140f7280768e4ad592309e9b7d6a26da P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:35.216014 26698 raft_consensus.cc:3037] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:35.218164 26565 raft_consensus.cc:2454] T 140f7280768e4ad592309e9b7d6a26da P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb27977c77564d38afe6e64c816eacd0 in term 1.
I20240502 21:35:35.218670 26363 leader_election.cc:302] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b1043ab39c04568814296d029600092, cb27977c77564d38afe6e64c816eacd0; no voters: 
I20240502 21:35:35.218991 27115 raft_consensus.cc:2781] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:35.219188 27115 raft_consensus.cc:686] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [term 1 LEADER]: Becoming Leader. State: Replica: cb27977c77564d38afe6e64c816eacd0, State: Running, Role: LEADER
I20240502 21:35:35.219178 26698 raft_consensus.cc:2454] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb27977c77564d38afe6e64c816eacd0 in term 1.
I20240502 21:35:35.219740 27115 consensus_queue.cc:237] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:35.223013 26283 catalog_manager.cc:5484] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb27977c77564d38afe6e64c816eacd0 (127.27.196.65). New cstate: current_term: 1 leader_uuid: "cb27977c77564d38afe6e64c816eacd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } health_report { overall_health: UNKNOWN } } }
I20240502 21:35:35.267537 27055 raft_consensus.cc:482] T 1331eade23b14aee86ceeb067c93a9d0 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:35:35.267777 27055 raft_consensus.cc:504] T 1331eade23b14aee86ceeb067c93a9d0 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:35.268535 27055 leader_election.cc:288] T 1331eade23b14aee86ceeb067c93a9d0 P 63266faf827046629e39afa765b144bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb27977c77564d38afe6e64c816eacd0 (127.27.196.65:36209), 5b1043ab39c04568814296d029600092 (127.27.196.66:41285)
I20240502 21:35:35.269244 26429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1331eade23b14aee86ceeb067c93a9d0" candidate_uuid: "63266faf827046629e39afa765b144bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb27977c77564d38afe6e64c816eacd0" is_pre_election: true
I20240502 21:35:35.269276 26565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1331eade23b14aee86ceeb067c93a9d0" candidate_uuid: "63266faf827046629e39afa765b144bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1043ab39c04568814296d029600092" is_pre_election: true
I20240502 21:35:35.269609 26565 raft_consensus.cc:2379] T 1331eade23b14aee86ceeb067c93a9d0 P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63266faf827046629e39afa765b144bc in current term 1: Already voted for candidate cb27977c77564d38afe6e64c816eacd0 in this term.
I20240502 21:35:35.270210 26632 leader_election.cc:302] T 1331eade23b14aee86ceeb067c93a9d0 P 63266faf827046629e39afa765b144bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63266faf827046629e39afa765b144bc; no voters: 5b1043ab39c04568814296d029600092, cb27977c77564d38afe6e64c816eacd0
I20240502 21:35:35.270617 27055 raft_consensus.cc:3037] T 1331eade23b14aee86ceeb067c93a9d0 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:35.274116 27055 raft_consensus.cc:2726] T 1331eade23b14aee86ceeb067c93a9d0 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:35:35.452199 27055 consensus_queue.cc:1035] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:35:35.455554 27055 consensus_queue.cc:1035] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:35:35.463198 27072 consensus_queue.cc:1035] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:35:35.474579 27045 consensus_queue.cc:1035] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:35:35.666124 27045 consensus_queue.cc:1035] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:35:35.673300 27045 consensus_queue.cc:1035] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:35:35.697091 27151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:35:35.697587 27151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:35:35.706871 27151 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'

INSERT report
    rows total: 2000
    time total: 247.434 ms
  time per row: 0.123717 ms
I20240502 21:35:36.135218 26283 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55362:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240502 21:35:36.136046 26283 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 21:35:36.149096 26410 tablet_service.cc:1467] Processing CreateTablet for tablet d5cf433249bc4c309c56c9d2e02efebe (DEFAULT_TABLE table=rebalance_test_table_2 [id=79000138ee6c4b96965a29a347143ffa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:36.149114 26544 tablet_service.cc:1467] Processing CreateTablet for tablet d5cf433249bc4c309c56c9d2e02efebe (DEFAULT_TABLE table=rebalance_test_table_2 [id=79000138ee6c4b96965a29a347143ffa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:36.149426 26676 tablet_service.cc:1467] Processing CreateTablet for tablet d5cf433249bc4c309c56c9d2e02efebe (DEFAULT_TABLE table=rebalance_test_table_2 [id=79000138ee6c4b96965a29a347143ffa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:36.149827 26544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5cf433249bc4c309c56c9d2e02efebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:36.149824 26410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5cf433249bc4c309c56c9d2e02efebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:36.150053 26676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5cf433249bc4c309c56c9d2e02efebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:36.150998 26542 tablet_service.cc:1467] Processing CreateTablet for tablet e5bca5fdd4d14b5baaa0b0cfc4754a23 (DEFAULT_TABLE table=rebalance_test_table_2 [id=79000138ee6c4b96965a29a347143ffa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:36.151641 26542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5bca5fdd4d14b5baaa0b0cfc4754a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:36.155756 26543 tablet_service.cc:1467] Processing CreateTablet for tablet fc2c1ffd7e084141a722d0a80febd74a (DEFAULT_TABLE table=rebalance_test_table_2 [id=79000138ee6c4b96965a29a347143ffa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:36.156268 26543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2c1ffd7e084141a722d0a80febd74a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:36.159976 26408 tablet_service.cc:1467] Processing CreateTablet for tablet fc2c1ffd7e084141a722d0a80febd74a (DEFAULT_TABLE table=rebalance_test_table_2 [id=79000138ee6c4b96965a29a347143ffa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:36.160804 26408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2c1ffd7e084141a722d0a80febd74a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:36.161034 26678 tablet_service.cc:1467] Processing CreateTablet for tablet fc2c1ffd7e084141a722d0a80febd74a (DEFAULT_TABLE table=rebalance_test_table_2 [id=79000138ee6c4b96965a29a347143ffa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:36.161604 27213 tablet_bootstrap.cc:492] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092: Bootstrap starting.
I20240502 21:35:36.161911 26677 tablet_service.cc:1467] Processing CreateTablet for tablet e5bca5fdd4d14b5baaa0b0cfc4754a23 (DEFAULT_TABLE table=rebalance_test_table_2 [id=79000138ee6c4b96965a29a347143ffa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:36.162568 26409 tablet_service.cc:1467] Processing CreateTablet for tablet e5bca5fdd4d14b5baaa0b0cfc4754a23 (DEFAULT_TABLE table=rebalance_test_table_2 [id=79000138ee6c4b96965a29a347143ffa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:36.163038 26409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5bca5fdd4d14b5baaa0b0cfc4754a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:36.163611 27213 tablet_bootstrap.cc:654] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:36.164846 26678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2c1ffd7e084141a722d0a80febd74a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:36.167356 26677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5bca5fdd4d14b5baaa0b0cfc4754a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:36.169345 27214 tablet_bootstrap.cc:492] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0: Bootstrap starting.
I20240502 21:35:36.172048 27214 tablet_bootstrap.cc:654] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:36.186578 27213 tablet_bootstrap.cc:492] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092: No bootstrap required, opened a new log
I20240502 21:35:36.187024 27213 ts_tablet_manager.cc:1367] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092: Time spent bootstrapping tablet: real 0.026s	user 0.008s	sys 0.016s
I20240502 21:35:36.190163 27213 raft_consensus.cc:348] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:36.190683 27216 tablet_bootstrap.cc:492] T d5cf433249bc4c309c56c9d2e02efebe P 63266faf827046629e39afa765b144bc: Bootstrap starting.
I20240502 21:35:36.190773 27213 raft_consensus.cc:374] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:36.190991 27213 raft_consensus.cc:729] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b1043ab39c04568814296d029600092, State: Initialized, Role: FOLLOWER
I20240502 21:35:36.191462 27213 consensus_queue.cc:260] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:36.193279 27213 ts_tablet_manager.cc:1398] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.003s
I20240502 21:35:36.194357 27213 tablet_bootstrap.cc:492] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092: Bootstrap starting.
I20240502 21:35:36.195312 27216 tablet_bootstrap.cc:654] T d5cf433249bc4c309c56c9d2e02efebe P 63266faf827046629e39afa765b144bc: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:36.197743 27213 tablet_bootstrap.cc:654] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:36.197820 27214 tablet_bootstrap.cc:492] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0: No bootstrap required, opened a new log
I20240502 21:35:36.198068 27214 ts_tablet_manager.cc:1367] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0: Time spent bootstrapping tablet: real 0.029s	user 0.005s	sys 0.020s
I20240502 21:35:36.199301 27214 raft_consensus.cc:348] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:36.199613 27214 raft_consensus.cc:374] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:36.199770 27214 raft_consensus.cc:729] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb27977c77564d38afe6e64c816eacd0, State: Initialized, Role: FOLLOWER
I20240502 21:35:36.200088 27214 consensus_queue.cc:260] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:36.201483 27214 ts_tablet_manager.cc:1398] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 21:35:36.201840 27214 tablet_bootstrap.cc:492] T fc2c1ffd7e084141a722d0a80febd74a P cb27977c77564d38afe6e64c816eacd0: Bootstrap starting.
I20240502 21:35:36.205502 27214 tablet_bootstrap.cc:654] T fc2c1ffd7e084141a722d0a80febd74a P cb27977c77564d38afe6e64c816eacd0: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:36.211431 27216 tablet_bootstrap.cc:492] T d5cf433249bc4c309c56c9d2e02efebe P 63266faf827046629e39afa765b144bc: No bootstrap required, opened a new log
I20240502 21:35:36.211913 27216 ts_tablet_manager.cc:1367] T d5cf433249bc4c309c56c9d2e02efebe P 63266faf827046629e39afa765b144bc: Time spent bootstrapping tablet: real 0.021s	user 0.009s	sys 0.000s
I20240502 21:35:36.215670 27216 raft_consensus.cc:348] T d5cf433249bc4c309c56c9d2e02efebe P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:36.216260 27216 raft_consensus.cc:374] T d5cf433249bc4c309c56c9d2e02efebe P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:36.216460 27216 raft_consensus.cc:729] T d5cf433249bc4c309c56c9d2e02efebe P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63266faf827046629e39afa765b144bc, State: Initialized, Role: FOLLOWER
I20240502 21:35:36.217175 27216 consensus_queue.cc:260] T d5cf433249bc4c309c56c9d2e02efebe P 63266faf827046629e39afa765b144bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:36.218539 27214 tablet_bootstrap.cc:492] T fc2c1ffd7e084141a722d0a80febd74a P cb27977c77564d38afe6e64c816eacd0: No bootstrap required, opened a new log
I20240502 21:35:36.218858 27214 ts_tablet_manager.cc:1367] T fc2c1ffd7e084141a722d0a80febd74a P cb27977c77564d38afe6e64c816eacd0: Time spent bootstrapping tablet: real 0.017s	user 0.010s	sys 0.005s
I20240502 21:35:36.219832 27214 raft_consensus.cc:348] T fc2c1ffd7e084141a722d0a80febd74a P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:36.221081 27216 ts_tablet_manager.cc:1398] T d5cf433249bc4c309c56c9d2e02efebe P 63266faf827046629e39afa765b144bc: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.000s
I20240502 21:35:36.221637 27216 tablet_bootstrap.cc:492] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc: Bootstrap starting.
I20240502 21:35:36.221607 27214 raft_consensus.cc:374] T fc2c1ffd7e084141a722d0a80febd74a P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:36.221799 27214 raft_consensus.cc:729] T fc2c1ffd7e084141a722d0a80febd74a P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb27977c77564d38afe6e64c816eacd0, State: Initialized, Role: FOLLOWER
I20240502 21:35:36.222100 27214 consensus_queue.cc:260] T fc2c1ffd7e084141a722d0a80febd74a P cb27977c77564d38afe6e64c816eacd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:36.222929 27213 tablet_bootstrap.cc:492] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092: No bootstrap required, opened a new log
I20240502 21:35:36.223181 27213 ts_tablet_manager.cc:1367] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092: Time spent bootstrapping tablet: real 0.029s	user 0.026s	sys 0.000s
I20240502 21:35:36.224151 27213 raft_consensus.cc:348] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:36.224422 27213 raft_consensus.cc:374] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:36.224509 27213 raft_consensus.cc:729] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b1043ab39c04568814296d029600092, State: Initialized, Role: FOLLOWER
I20240502 21:35:36.224637 27216 tablet_bootstrap.cc:654] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:36.224788 27213 consensus_queue.cc:260] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:36.226925 27214 ts_tablet_manager.cc:1398] T fc2c1ffd7e084141a722d0a80febd74a P cb27977c77564d38afe6e64c816eacd0: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.000s
I20240502 21:35:36.227313 27214 tablet_bootstrap.cc:492] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P cb27977c77564d38afe6e64c816eacd0: Bootstrap starting.
I20240502 21:35:36.229058 27214 tablet_bootstrap.cc:654] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P cb27977c77564d38afe6e64c816eacd0: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:36.230093 27213 ts_tablet_manager.cc:1398] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.000s
I20240502 21:35:36.230465 27213 tablet_bootstrap.cc:492] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092: Bootstrap starting.
I20240502 21:35:36.235539 27213 tablet_bootstrap.cc:654] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:36.235805 27216 tablet_bootstrap.cc:492] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc: No bootstrap required, opened a new log
I20240502 21:35:36.236147 27216 ts_tablet_manager.cc:1367] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc: Time spent bootstrapping tablet: real 0.015s	user 0.007s	sys 0.004s
I20240502 21:35:36.237346 27216 raft_consensus.cc:348] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:36.237694 27216 raft_consensus.cc:374] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:36.237778 27216 raft_consensus.cc:729] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63266faf827046629e39afa765b144bc, State: Initialized, Role: FOLLOWER
I20240502 21:35:36.238072 27216 consensus_queue.cc:260] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:36.238343 27214 tablet_bootstrap.cc:492] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P cb27977c77564d38afe6e64c816eacd0: No bootstrap required, opened a new log
I20240502 21:35:36.238548 27214 ts_tablet_manager.cc:1367] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P cb27977c77564d38afe6e64c816eacd0: Time spent bootstrapping tablet: real 0.011s	user 0.004s	sys 0.000s
I20240502 21:35:36.238970 27216 ts_tablet_manager.cc:1398] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240502 21:35:36.239181 27213 tablet_bootstrap.cc:492] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092: No bootstrap required, opened a new log
I20240502 21:35:36.239413 27213 ts_tablet_manager.cc:1367] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.001s
I20240502 21:35:36.239583 27216 tablet_bootstrap.cc:492] T fc2c1ffd7e084141a722d0a80febd74a P 63266faf827046629e39afa765b144bc: Bootstrap starting.
I20240502 21:35:36.239614 27214 raft_consensus.cc:348] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:36.239864 27214 raft_consensus.cc:374] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:36.239961 27214 raft_consensus.cc:729] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb27977c77564d38afe6e64c816eacd0, State: Initialized, Role: FOLLOWER
I20240502 21:35:36.240224 27214 consensus_queue.cc:260] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P cb27977c77564d38afe6e64c816eacd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:36.240393 27213 raft_consensus.cc:348] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:36.240600 27213 raft_consensus.cc:374] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:36.240692 27213 raft_consensus.cc:729] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b1043ab39c04568814296d029600092, State: Initialized, Role: FOLLOWER
I20240502 21:35:36.240911 27214 ts_tablet_manager.cc:1398] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P cb27977c77564d38afe6e64c816eacd0: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:35:36.240942 27213 consensus_queue.cc:260] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:36.241554 27213 ts_tablet_manager.cc:1398] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 21:35:36.241724 27216 tablet_bootstrap.cc:654] T fc2c1ffd7e084141a722d0a80febd74a P 63266faf827046629e39afa765b144bc: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:36.245648 27216 tablet_bootstrap.cc:492] T fc2c1ffd7e084141a722d0a80febd74a P 63266faf827046629e39afa765b144bc: No bootstrap required, opened a new log
I20240502 21:35:36.245857 27216 ts_tablet_manager.cc:1367] T fc2c1ffd7e084141a722d0a80febd74a P 63266faf827046629e39afa765b144bc: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.002s
I20240502 21:35:36.246748 27216 raft_consensus.cc:348] T fc2c1ffd7e084141a722d0a80febd74a P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:36.246976 27216 raft_consensus.cc:374] T fc2c1ffd7e084141a722d0a80febd74a P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:36.247094 27216 raft_consensus.cc:729] T fc2c1ffd7e084141a722d0a80febd74a P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63266faf827046629e39afa765b144bc, State: Initialized, Role: FOLLOWER
I20240502 21:35:36.247368 27216 consensus_queue.cc:260] T fc2c1ffd7e084141a722d0a80febd74a P 63266faf827046629e39afa765b144bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:36.247886 27216 ts_tablet_manager.cc:1398] T fc2c1ffd7e084141a722d0a80febd74a P 63266faf827046629e39afa765b144bc: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 21:35:36.309898 27159 raft_consensus.cc:482] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:35:36.310138 27159 raft_consensus.cc:504] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:36.311532 27159 leader_election.cc:288] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb27977c77564d38afe6e64c816eacd0 (127.27.196.65:36209), 63266faf827046629e39afa765b144bc (127.27.196.67:39203)
I20240502 21:35:36.312235 26697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5bca5fdd4d14b5baaa0b0cfc4754a23" candidate_uuid: "5b1043ab39c04568814296d029600092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63266faf827046629e39afa765b144bc" is_pre_election: true
I20240502 21:35:36.312278 26429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5bca5fdd4d14b5baaa0b0cfc4754a23" candidate_uuid: "5b1043ab39c04568814296d029600092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb27977c77564d38afe6e64c816eacd0" is_pre_election: true
I20240502 21:35:36.312570 26697 raft_consensus.cc:2454] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b1043ab39c04568814296d029600092 in term 0.
I20240502 21:35:36.312608 26429 raft_consensus.cc:2454] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b1043ab39c04568814296d029600092 in term 0.
I20240502 21:35:36.313097 26499 leader_election.cc:302] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b1043ab39c04568814296d029600092, 63266faf827046629e39afa765b144bc; no voters: 
I20240502 21:35:36.313431 27227 raft_consensus.cc:2781] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:35:36.313570 27227 raft_consensus.cc:482] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:35:36.313735 27227 raft_consensus.cc:3037] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:36.314574 27072 raft_consensus.cc:482] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:35:36.314711 27072 raft_consensus.cc:504] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:36.315412 27072 leader_election.cc:288] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b1043ab39c04568814296d029600092 (127.27.196.66:41285), 63266faf827046629e39afa765b144bc (127.27.196.67:39203)
I20240502 21:35:36.316092 26565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5cf433249bc4c309c56c9d2e02efebe" candidate_uuid: "cb27977c77564d38afe6e64c816eacd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1043ab39c04568814296d029600092" is_pre_election: true
I20240502 21:35:36.316102 26697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5cf433249bc4c309c56c9d2e02efebe" candidate_uuid: "cb27977c77564d38afe6e64c816eacd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63266faf827046629e39afa765b144bc" is_pre_election: true
I20240502 21:35:36.316323 26697 raft_consensus.cc:2454] T d5cf433249bc4c309c56c9d2e02efebe P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb27977c77564d38afe6e64c816eacd0 in term 0.
I20240502 21:35:36.316339 26565 raft_consensus.cc:2454] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb27977c77564d38afe6e64c816eacd0 in term 0.
I20240502 21:35:36.316825 26364 leader_election.cc:302] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63266faf827046629e39afa765b144bc, cb27977c77564d38afe6e64c816eacd0; no voters: 
I20240502 21:35:36.317153 27072 raft_consensus.cc:2781] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:35:36.316951 27227 raft_consensus.cc:504] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:36.317320 27072 raft_consensus.cc:482] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:35:36.317449 27072 raft_consensus.cc:3037] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:36.317848 27227 leader_election.cc:288] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [CANDIDATE]: Term 1 election: Requested vote from peers cb27977c77564d38afe6e64c816eacd0 (127.27.196.65:36209), 63266faf827046629e39afa765b144bc (127.27.196.67:39203)
I20240502 21:35:36.318416 26697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5bca5fdd4d14b5baaa0b0cfc4754a23" candidate_uuid: "5b1043ab39c04568814296d029600092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63266faf827046629e39afa765b144bc"
I20240502 21:35:36.318467 26429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5bca5fdd4d14b5baaa0b0cfc4754a23" candidate_uuid: "5b1043ab39c04568814296d029600092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb27977c77564d38afe6e64c816eacd0"
I20240502 21:35:36.318650 26697 raft_consensus.cc:3037] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:36.318694 26429 raft_consensus.cc:3037] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:36.320811 26697 raft_consensus.cc:2454] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b1043ab39c04568814296d029600092 in term 1.
I20240502 21:35:36.320751 27072 raft_consensus.cc:504] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:36.321333 26499 leader_election.cc:302] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b1043ab39c04568814296d029600092, 63266faf827046629e39afa765b144bc; no voters: 
I20240502 21:35:36.321866 26429 raft_consensus.cc:2454] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b1043ab39c04568814296d029600092 in term 1.
I20240502 21:35:36.321982 27227 raft_consensus.cc:2781] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:36.322068 27072 leader_election.cc:288] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [CANDIDATE]: Term 1 election: Requested vote from peers 5b1043ab39c04568814296d029600092 (127.27.196.66:41285), 63266faf827046629e39afa765b144bc (127.27.196.67:39203)
I20240502 21:35:36.322669 26565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5cf433249bc4c309c56c9d2e02efebe" candidate_uuid: "cb27977c77564d38afe6e64c816eacd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1043ab39c04568814296d029600092"
I20240502 21:35:36.322821 27227 raft_consensus.cc:686] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [term 1 LEADER]: Becoming Leader. State: Replica: 5b1043ab39c04568814296d029600092, State: Running, Role: LEADER
I20240502 21:35:36.323056 26565 raft_consensus.cc:3037] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:36.323307 26697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5cf433249bc4c309c56c9d2e02efebe" candidate_uuid: "cb27977c77564d38afe6e64c816eacd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63266faf827046629e39afa765b144bc"
I20240502 21:35:36.323598 26697 raft_consensus.cc:3037] T d5cf433249bc4c309c56c9d2e02efebe P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:36.323454 27227 consensus_queue.cc:237] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:36.326310 26697 raft_consensus.cc:2454] T d5cf433249bc4c309c56c9d2e02efebe P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb27977c77564d38afe6e64c816eacd0 in term 1.
I20240502 21:35:36.327309 26364 leader_election.cc:302] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63266faf827046629e39afa765b144bc, cb27977c77564d38afe6e64c816eacd0; no voters: 
I20240502 21:35:36.327917 27072 raft_consensus.cc:2781] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:36.328193 27072 raft_consensus.cc:686] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [term 1 LEADER]: Becoming Leader. State: Replica: cb27977c77564d38afe6e64c816eacd0, State: Running, Role: LEADER
I20240502 21:35:36.328545 26565 raft_consensus.cc:2454] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb27977c77564d38afe6e64c816eacd0 in term 1.
I20240502 21:35:36.328559 27072 consensus_queue.cc:237] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:36.329881 26283 catalog_manager.cc:5484] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b1043ab39c04568814296d029600092 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "5b1043ab39c04568814296d029600092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } health_report { overall_health: UNKNOWN } } }
I20240502 21:35:36.332490 26284 catalog_manager.cc:5484] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb27977c77564d38afe6e64c816eacd0 (127.27.196.65). New cstate: current_term: 1 leader_uuid: "cb27977c77564d38afe6e64c816eacd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } health_report { overall_health: UNKNOWN } } }
I20240502 21:35:36.545428 27227 raft_consensus.cc:482] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:35:36.545681 27227 raft_consensus.cc:504] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:36.546581 27227 leader_election.cc:288] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb27977c77564d38afe6e64c816eacd0 (127.27.196.65:36209), 63266faf827046629e39afa765b144bc (127.27.196.67:39203)
I20240502 21:35:36.547330 26429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2c1ffd7e084141a722d0a80febd74a" candidate_uuid: "5b1043ab39c04568814296d029600092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb27977c77564d38afe6e64c816eacd0" is_pre_election: true
I20240502 21:35:36.547319 26697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2c1ffd7e084141a722d0a80febd74a" candidate_uuid: "5b1043ab39c04568814296d029600092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63266faf827046629e39afa765b144bc" is_pre_election: true
I20240502 21:35:36.547569 26697 raft_consensus.cc:2454] T fc2c1ffd7e084141a722d0a80febd74a P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b1043ab39c04568814296d029600092 in term 0.
I20240502 21:35:36.547587 26429 raft_consensus.cc:2454] T fc2c1ffd7e084141a722d0a80febd74a P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b1043ab39c04568814296d029600092 in term 0.
I20240502 21:35:36.548053 26498 leader_election.cc:302] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b1043ab39c04568814296d029600092, cb27977c77564d38afe6e64c816eacd0; no voters: 
I20240502 21:35:36.548385 27227 raft_consensus.cc:2781] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:35:36.548496 27227 raft_consensus.cc:482] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:35:36.548564 27227 raft_consensus.cc:3037] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:36.550521 27227 raft_consensus.cc:504] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:36.551259 27227 leader_election.cc:288] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [CANDIDATE]: Term 1 election: Requested vote from peers cb27977c77564d38afe6e64c816eacd0 (127.27.196.65:36209), 63266faf827046629e39afa765b144bc (127.27.196.67:39203)
I20240502 21:35:36.551978 26429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2c1ffd7e084141a722d0a80febd74a" candidate_uuid: "5b1043ab39c04568814296d029600092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb27977c77564d38afe6e64c816eacd0"
I20240502 21:35:36.552016 26697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2c1ffd7e084141a722d0a80febd74a" candidate_uuid: "5b1043ab39c04568814296d029600092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63266faf827046629e39afa765b144bc"
I20240502 21:35:36.552263 26429 raft_consensus.cc:3037] T fc2c1ffd7e084141a722d0a80febd74a P cb27977c77564d38afe6e64c816eacd0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:36.552266 26697 raft_consensus.cc:3037] T fc2c1ffd7e084141a722d0a80febd74a P 63266faf827046629e39afa765b144bc [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:36.554404 26697 raft_consensus.cc:2454] T fc2c1ffd7e084141a722d0a80febd74a P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b1043ab39c04568814296d029600092 in term 1.
I20240502 21:35:36.554677 26429 raft_consensus.cc:2454] T fc2c1ffd7e084141a722d0a80febd74a P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b1043ab39c04568814296d029600092 in term 1.
I20240502 21:35:36.555007 26499 leader_election.cc:302] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b1043ab39c04568814296d029600092, 63266faf827046629e39afa765b144bc; no voters: 
I20240502 21:35:36.555702 27227 raft_consensus.cc:2781] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:36.555860 27227 raft_consensus.cc:686] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [term 1 LEADER]: Becoming Leader. State: Replica: 5b1043ab39c04568814296d029600092, State: Running, Role: LEADER
I20240502 21:35:36.556192 27227 consensus_queue.cc:237] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } }
I20240502 21:35:36.560405 26284 catalog_manager.cc:5484] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b1043ab39c04568814296d029600092 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "5b1043ab39c04568814296d029600092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } health_report { overall_health: UNKNOWN } } }
I20240502 21:35:36.738616 27227 consensus_queue.cc:1035] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:35:36.756824 27159 consensus_queue.cc:1035] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:35:36.851408 27072 consensus_queue.cc:1035] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:35:36.858076 27072 consensus_queue.cc:1035] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:35:36.935225 27159 consensus_queue.cc:1035] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:35:36.944293 27235 consensus_queue.cc:1035] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:35:37.104173 27231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:35:37.104544 27231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:35:37.112596 27231 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'

INSERT report
    rows total: 2000
    time total: 293.822 ms
  time per row: 0.146911 ms
I20240502 21:35:37.574489 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.69:43269
--local_ip_for_outbound_sockets=127.27.196.69
--tserver_master_addrs=127.27.196.126:36809
--webserver_port=37245
--webserver_interface=127.27.196.69
--builtin_ntp_servers=127.27.196.84:36263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:35:37.974732 27291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:35:37.975096 27291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:35:37.975266 27291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:35:37.984997 27291 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:35:37.985170 27291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:35:37.985343 27291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.69
I20240502 21:35:37.996687 27291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:36263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.69:43269
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.27.196.69
--webserver_port=37245
--tserver_master_addrs=127.27.196.126:36809
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.69
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:35:37.998129 27291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:35:38.000052 27291 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:35:38.016458 27296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:38.017050 27297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:38.017143 27299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:35:38.017676 27291 server_base.cc:1034] running on GCE node
I20240502 21:35:38.018414 27291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:35:38.019542 27291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:35:38.020787 27291 hybrid_clock.cc:648] HybridClock initialized: now 1714685738020734 us; error 54 us; skew 500 ppm
I20240502 21:35:38.021145 27291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:35:38.023823 27291 webserver.cc:460] Webserver started at http://127.27.196.69:37245/ using document root <none> and password file <none>
I20240502 21:35:38.025074 27291 fs_manager.cc:359] Metadata directory not provided
I20240502 21:35:38.025348 27291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:35:38.033017 27291 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240502 21:35:38.036324 27305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:38.038028 27291 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240502 21:35:38.038210 27291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c609c27f61b9436d95dc53deaa21a3b0"
format_stamp: "Formatted at 2024-05-02 21:35:29 on dist-test-slave-3bs1"
I20240502 21:35:38.039093 27291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:35:38.055918 27291 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:35:38.056807 27291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:35:38.057112 27291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:35:38.058218 27291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:35:38.060176 27291 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:35:38.060268 27291 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:38.060407 27291 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:35:38.060459 27291 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:38.098199 27291 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.69:43269
I20240502 21:35:38.098239 27420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.69:43269 every 8 connection(s)
I20240502 21:35:38.099895 27291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:35:38.109370 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 27291
I20240502 21:35:38.110088 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.68:35953
--local_ip_for_outbound_sockets=127.27.196.68
--tserver_master_addrs=127.27.196.126:36809
--webserver_port=40663
--webserver_interface=127.27.196.68
--builtin_ntp_servers=127.27.196.84:36263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:35:38.116065 27421 heartbeater.cc:347] Connected to a master server at 127.27.196.126:36809
I20240502 21:35:38.116467 27421 heartbeater.cc:464] Registering TS with master...
I20240502 21:35:38.117516 27421 heartbeater.cc:511] Master 127.27.196.126:36809 requested a full tablet report, sending...
I20240502 21:35:38.119612 26286 ts_manager.cc:194] Re-registered known tserver with Master: c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69:43269)
I20240502 21:35:38.121071 26286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.69:52183
W20240502 21:35:38.527060 27425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:35:38.527557 27425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:35:38.527738 27425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:35:38.538441 27425 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:35:38.538594 27425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:35:38.538785 27425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.68
I20240502 21:35:38.550825 27425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:36263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.68:35953
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.27.196.68
--webserver_port=40663
--tserver_master_addrs=127.27.196.126:36809
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.68
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:35:38.552340 27425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:35:38.554280 27425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:35:38.568812 27431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:38.569437 27432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:38.569983 27434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:35:38.570257 27425 server_base.cc:1034] running on GCE node
I20240502 21:35:38.571013 27425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:35:38.572324 27425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:35:38.573601 27425 hybrid_clock.cc:648] HybridClock initialized: now 1714685738573540 us; error 60 us; skew 500 ppm
I20240502 21:35:38.574013 27425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:35:38.576845 27425 webserver.cc:460] Webserver started at http://127.27.196.68:40663/ using document root <none> and password file <none>
I20240502 21:35:38.577737 27425 fs_manager.cc:359] Metadata directory not provided
I20240502 21:35:38.577886 27425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:35:38.584326 27425 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.001s
I20240502 21:35:38.587567 27440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:38.589020 27425 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240502 21:35:38.589226 27425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "27be1297e75b4e20ba297e7332a98f92"
format_stamp: "Formatted at 2024-05-02 21:35:28 on dist-test-slave-3bs1"
I20240502 21:35:38.590206 27425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:35:38.619201 27425 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:35:38.620170 27425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:35:38.620502 27425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:35:38.621675 27425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:35:38.623762 27425 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:35:38.623863 27425 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:38.624017 27425 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:35:38.624076 27425 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:38.664383 27425 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.68:35953
I20240502 21:35:38.664435 27556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.68:35953 every 8 connection(s)
I20240502 21:35:38.665979 27425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:35:38.668357 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 27425
I20240502 21:35:38.683465 27557 heartbeater.cc:347] Connected to a master server at 127.27.196.126:36809
I20240502 21:35:38.683866 27557 heartbeater.cc:464] Registering TS with master...
I20240502 21:35:38.685178 27557 heartbeater.cc:511] Master 127.27.196.126:36809 requested a full tablet report, sending...
I20240502 21:35:38.687916 26286 ts_manager.cc:194] Re-registered known tserver with Master: 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68:35953)
I20240502 21:35:38.688970 26286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.68:59401
I20240502 21:35:39.124307 27421 heartbeater.cc:503] Master 127.27.196.126:36809 was elected leader, sending a full tablet report...
I20240502 21:35:39.286271 26543 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:35:39.300441 26678 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:39.304307 26409 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:35:39.315230 27490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:35:39.327530 27354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:35:39.520045 27490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:35:39.525956 27354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:35:39.527846 26678 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:39.532325 26543 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:35:39.549317 26409 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:35:39.693336 27557 heartbeater.cc:503] Master 127.27.196.126:36809 was elected leader, sending a full tablet report...
I20240502 21:35:39.760183 27354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:35:39.761133 27490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:35:39.761719 26543 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:35:39.777390 26409 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:35:39.790218 26678 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:39.992533 27490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:35:40.010953 26409 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:35:40.012451 26543 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:35:40.015841 26678 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:40.023280 27354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:35:40.124536 26429 consensus_queue.cc:237] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } }
I20240502 21:35:40.128702 26698 raft_consensus.cc:1261] T d5cf433249bc4c309c56c9d2e02efebe P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:35:40.129189 26564 raft_consensus.cc:1261] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:35:40.129438 27426 consensus_queue.cc:1035] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:35:40.130087 27653 consensus_queue.cc:1035] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:35:40.134579 27426 raft_consensus.cc:2932] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } } }
I20240502 21:35:40.135076 26564 raft_consensus.cc:2932] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } } }
I20240502 21:35:40.136278 26698 raft_consensus.cc:2932] T d5cf433249bc4c309c56c9d2e02efebe P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } } }
W20240502 21:35:40.142079 26363 consensus_peers.cc:485] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 -> Peer 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68:35953): Couldn't send request to peer 27be1297e75b4e20ba297e7332a98f92. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5cf433249bc4c309c56c9d2e02efebe. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:40.144088 26286 catalog_manager.cc:5484] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092 reported cstate change: config changed from index -1 to 4, NON_VOTER 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) added. New cstate: current_term: 1 leader_uuid: "cb27977c77564d38afe6e64c816eacd0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } } }
I20240502 21:35:40.163097 26429 consensus_queue.cc:237] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: true } }
I20240502 21:35:40.166989 26698 raft_consensus.cc:1261] T 1331eade23b14aee86ceeb067c93a9d0 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:35:40.167970 27711 consensus_queue.cc:1035] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:35:40.168251 26564 raft_consensus.cc:1261] T 1331eade23b14aee86ceeb067c93a9d0 P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:35:40.171087 27711 consensus_queue.cc:1035] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:35:40.180166 26362 consensus_peers.cc:485] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 -> Peer c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69:43269): Couldn't send request to peer c609c27f61b9436d95dc53deaa21a3b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1331eade23b14aee86ceeb067c93a9d0. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:40.181274 27653 raft_consensus.cc:2932] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: true } } }
I20240502 21:35:40.182327 26697 raft_consensus.cc:2932] T 1331eade23b14aee86ceeb067c93a9d0 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: true } } }
I20240502 21:35:40.189309 26284 catalog_manager.cc:5484] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 reported cstate change: config changed from index -1 to 4, NON_VOTER c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) added. New cstate: current_term: 1 leader_uuid: "cb27977c77564d38afe6e64c816eacd0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:35:40.200315 26564 raft_consensus.cc:2932] T 1331eade23b14aee86ceeb067c93a9d0 P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: true } } }
I20240502 21:35:40.228147 26697 consensus_queue.cc:237] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } }
I20240502 21:35:40.232635 26429 raft_consensus.cc:1261] T a3a929c153c348ea8a0b4e7c2cf838f6 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Refusing update from remote peer 63266faf827046629e39afa765b144bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:35:40.232913 26564 raft_consensus.cc:1261] T a3a929c153c348ea8a0b4e7c2cf838f6 P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Refusing update from remote peer 63266faf827046629e39afa765b144bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:35:40.233714 27724 consensus_queue.cc:1035] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:35:40.234181 27572 consensus_queue.cc:1035] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:35:40.241077 26632 consensus_peers.cc:485] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc -> Peer 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68:35953): Couldn't send request to peer 27be1297e75b4e20ba297e7332a98f92. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3a929c153c348ea8a0b4e7c2cf838f6. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:40.245290 27724 raft_consensus.cc:2932] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } } }
I20240502 21:35:40.246435 26564 raft_consensus.cc:2932] T a3a929c153c348ea8a0b4e7c2cf838f6 P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } } }
I20240502 21:35:40.247291 26429 raft_consensus.cc:2932] T a3a929c153c348ea8a0b4e7c2cf838f6 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } } }
I20240502 21:35:40.253296 26287 catalog_manager.cc:5484] T a3a929c153c348ea8a0b4e7c2cf838f6 P 5b1043ab39c04568814296d029600092 reported cstate change: config changed from index -1 to 4, NON_VOTER 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) added. New cstate: current_term: 1 leader_uuid: "63266faf827046629e39afa765b144bc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } } }
I20240502 21:35:40.271040 26429 consensus_queue.cc:237] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: true } }
I20240502 21:35:40.273891 26697 raft_consensus.cc:1261] T c06cdd9252864256a6870171f0f4c09c P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:35:40.273900 26564 raft_consensus.cc:1261] T c06cdd9252864256a6870171f0f4c09c P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:35:40.274106 26362 consensus_peers.cc:485] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 -> Peer c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69:43269): Couldn't send request to peer c609c27f61b9436d95dc53deaa21a3b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c06cdd9252864256a6870171f0f4c09c. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:40.274596 27711 consensus_queue.cc:1035] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:35:40.274921 27426 consensus_queue.cc:1035] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:35:40.277810 27711 raft_consensus.cc:2932] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: true } } }
I20240502 21:35:40.278610 26697 raft_consensus.cc:2932] T c06cdd9252864256a6870171f0f4c09c P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: true } } }
I20240502 21:35:40.278962 26564 raft_consensus.cc:2932] T c06cdd9252864256a6870171f0f4c09c P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: true } } }
I20240502 21:35:40.285108 26282 catalog_manager.cc:5484] T c06cdd9252864256a6870171f0f4c09c P 5b1043ab39c04568814296d029600092 reported cstate change: config changed from index -1 to 4, NON_VOTER c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) added. New cstate: current_term: 1 leader_uuid: "cb27977c77564d38afe6e64c816eacd0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: true } } }
I20240502 21:35:40.301040 26429 consensus_queue.cc:237] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } }
I20240502 21:35:40.304571 26564 raft_consensus.cc:1261] T 140f7280768e4ad592309e9b7d6a26da P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:35:40.304756 26697 raft_consensus.cc:1261] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:35:40.304737 26363 consensus_peers.cc:485] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 -> Peer 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68:35953): Couldn't send request to peer 27be1297e75b4e20ba297e7332a98f92. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 140f7280768e4ad592309e9b7d6a26da. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:40.305485 27426 consensus_queue.cc:1035] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:35:40.305843 27653 consensus_queue.cc:1035] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:35:40.309391 27711 raft_consensus.cc:2932] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } } }
I20240502 21:35:40.310313 26697 raft_consensus.cc:2932] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } } }
I20240502 21:35:40.310443 26564 raft_consensus.cc:2932] T 140f7280768e4ad592309e9b7d6a26da P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } } }
I20240502 21:35:40.314442 26287 catalog_manager.cc:5484] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 reported cstate change: config changed from index -1 to 4, NON_VOTER 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) added. New cstate: current_term: 1 leader_uuid: "cb27977c77564d38afe6e64c816eacd0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:35:40.333905 26564 consensus_queue.cc:237] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: true } }
I20240502 21:35:40.338035 26429 raft_consensus.cc:1261] T fc2c1ffd7e084141a722d0a80febd74a P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Refusing update from remote peer 5b1043ab39c04568814296d029600092: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:35:40.339336 27759 consensus_queue.cc:1035] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:35:40.339848 26697 raft_consensus.cc:1261] T fc2c1ffd7e084141a722d0a80febd74a P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Refusing update from remote peer 5b1043ab39c04568814296d029600092: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:35:40.340749 27759 consensus_queue.cc:1035] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:35:40.345891 27759 raft_consensus.cc:2932] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: true } } }
I20240502 21:35:40.346647 26697 raft_consensus.cc:2932] T fc2c1ffd7e084141a722d0a80febd74a P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: true } } }
W20240502 21:35:40.348428 26497 consensus_peers.cc:485] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 -> Peer c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69:43269): Couldn't send request to peer c609c27f61b9436d95dc53deaa21a3b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc2c1ffd7e084141a722d0a80febd74a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:40.349454 26430 raft_consensus.cc:2932] T fc2c1ffd7e084141a722d0a80febd74a P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: true } } }
I20240502 21:35:40.353296 26286 catalog_manager.cc:5484] T fc2c1ffd7e084141a722d0a80febd74a P 63266faf827046629e39afa765b144bc reported cstate change: config changed from index -1 to 4, NON_VOTER c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) added. New cstate: current_term: 1 leader_uuid: "5b1043ab39c04568814296d029600092" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: true } } }
I20240502 21:35:40.373818 26564 consensus_queue.cc:237] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } }
I20240502 21:35:40.377138 26697 raft_consensus.cc:1261] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Refusing update from remote peer 5b1043ab39c04568814296d029600092: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:35:40.377869 26429 raft_consensus.cc:1261] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Refusing update from remote peer 5b1043ab39c04568814296d029600092: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:35:40.377873 27763 consensus_queue.cc:1035] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:35:40.378767 27763 consensus_queue.cc:1035] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:35:40.386643 26498 consensus_peers.cc:485] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 -> Peer 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68:35953): Couldn't send request to peer 27be1297e75b4e20ba297e7332a98f92. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8747e6794c084510a5b9e1f34c15c733. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:40.389161 27227 raft_consensus.cc:2932] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } } }
I20240502 21:35:40.392941 26697 raft_consensus.cc:2932] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } } }
I20240502 21:35:40.394320 26429 raft_consensus.cc:2932] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } } }
I20240502 21:35:40.395972 26284 catalog_manager.cc:5484] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 reported cstate change: config changed from index -1 to 4, NON_VOTER 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) added. New cstate: current_term: 1 leader_uuid: "5b1043ab39c04568814296d029600092" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:35:40.536105 27810 ts_tablet_manager.cc:902] T d5cf433249bc4c309c56c9d2e02efebe P 27be1297e75b4e20ba297e7332a98f92: Initiating tablet copy from peer cb27977c77564d38afe6e64c816eacd0 (127.27.196.65:36209)
I20240502 21:35:40.537992 27810 tablet_copy_client.cc:308] T d5cf433249bc4c309c56c9d2e02efebe P 27be1297e75b4e20ba297e7332a98f92: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:36209
I20240502 21:35:40.545563 26450 tablet_copy_service.cc:140] P cb27977c77564d38afe6e64c816eacd0: Received BeginTabletCopySession request for tablet d5cf433249bc4c309c56c9d2e02efebe from peer 27be1297e75b4e20ba297e7332a98f92 ({username='slave'} at 127.27.196.68:48789)
I20240502 21:35:40.545866 26450 tablet_copy_service.cc:161] P cb27977c77564d38afe6e64c816eacd0: Beginning new tablet copy session on tablet d5cf433249bc4c309c56c9d2e02efebe from peer 27be1297e75b4e20ba297e7332a98f92 at {username='slave'} at 127.27.196.68:48789: session id = 27be1297e75b4e20ba297e7332a98f92-d5cf433249bc4c309c56c9d2e02efebe
I20240502 21:35:40.549798 26450 tablet_copy_source_session.cc:205] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:35:40.555241 27810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5cf433249bc4c309c56c9d2e02efebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:40.567816 27810 tablet_copy_client.cc:783] T d5cf433249bc4c309c56c9d2e02efebe P 27be1297e75b4e20ba297e7332a98f92: tablet copy: Starting download of 0 data blocks...
I20240502 21:35:40.568275 27810 tablet_copy_client.cc:647] T d5cf433249bc4c309c56c9d2e02efebe P 27be1297e75b4e20ba297e7332a98f92: tablet copy: Starting download of 1 WAL segments...
I20240502 21:35:40.571826 27810 tablet_copy_client.cc:523] T d5cf433249bc4c309c56c9d2e02efebe P 27be1297e75b4e20ba297e7332a98f92: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:35:40.575713 27810 tablet_bootstrap.cc:492] T d5cf433249bc4c309c56c9d2e02efebe P 27be1297e75b4e20ba297e7332a98f92: Bootstrap starting.
I20240502 21:35:40.586699 27810 log.cc:824] T d5cf433249bc4c309c56c9d2e02efebe P 27be1297e75b4e20ba297e7332a98f92: Log is configured to *not* fsync() on all Append() calls
I20240502 21:35:40.618577 27817 ts_tablet_manager.cc:902] T 1331eade23b14aee86ceeb067c93a9d0 P c609c27f61b9436d95dc53deaa21a3b0: Initiating tablet copy from peer cb27977c77564d38afe6e64c816eacd0 (127.27.196.65:36209)
I20240502 21:35:40.620601 27817 tablet_copy_client.cc:308] T 1331eade23b14aee86ceeb067c93a9d0 P c609c27f61b9436d95dc53deaa21a3b0: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:36209
I20240502 21:35:40.622330 27810 tablet_bootstrap.cc:492] T d5cf433249bc4c309c56c9d2e02efebe P 27be1297e75b4e20ba297e7332a98f92: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:35:40.622854 27810 tablet_bootstrap.cc:492] T d5cf433249bc4c309c56c9d2e02efebe P 27be1297e75b4e20ba297e7332a98f92: Bootstrap complete.
I20240502 21:35:40.623713 27810 ts_tablet_manager.cc:1367] T d5cf433249bc4c309c56c9d2e02efebe P 27be1297e75b4e20ba297e7332a98f92: Time spent bootstrapping tablet: real 0.048s	user 0.039s	sys 0.009s
I20240502 21:35:40.627972 26450 tablet_copy_service.cc:140] P cb27977c77564d38afe6e64c816eacd0: Received BeginTabletCopySession request for tablet 1331eade23b14aee86ceeb067c93a9d0 from peer c609c27f61b9436d95dc53deaa21a3b0 ({username='slave'} at 127.27.196.69:44723)
I20240502 21:35:40.628214 26450 tablet_copy_service.cc:161] P cb27977c77564d38afe6e64c816eacd0: Beginning new tablet copy session on tablet 1331eade23b14aee86ceeb067c93a9d0 from peer c609c27f61b9436d95dc53deaa21a3b0 at {username='slave'} at 127.27.196.69:44723: session id = c609c27f61b9436d95dc53deaa21a3b0-1331eade23b14aee86ceeb067c93a9d0
I20240502 21:35:40.629907 27810 raft_consensus.cc:348] T d5cf433249bc4c309c56c9d2e02efebe P 27be1297e75b4e20ba297e7332a98f92 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } }
I20240502 21:35:40.630605 26450 tablet_copy_source_session.cc:205] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:35:40.630589 27810 raft_consensus.cc:729] T d5cf433249bc4c309c56c9d2e02efebe P 27be1297e75b4e20ba297e7332a98f92 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 27be1297e75b4e20ba297e7332a98f92, State: Initialized, Role: LEARNER
I20240502 21:35:40.631481 27810 consensus_queue.cc:260] T d5cf433249bc4c309c56c9d2e02efebe P 27be1297e75b4e20ba297e7332a98f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } }
I20240502 21:35:40.633500 27810 ts_tablet_manager.cc:1398] T d5cf433249bc4c309c56c9d2e02efebe P 27be1297e75b4e20ba297e7332a98f92: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.000s
I20240502 21:35:40.634636 26450 tablet_copy_service.cc:342] P cb27977c77564d38afe6e64c816eacd0: Request end of tablet copy session 27be1297e75b4e20ba297e7332a98f92-d5cf433249bc4c309c56c9d2e02efebe received from {username='slave'} at 127.27.196.68:48789
I20240502 21:35:40.634814 26450 tablet_copy_service.cc:433] P cb27977c77564d38afe6e64c816eacd0: ending tablet copy session 27be1297e75b4e20ba297e7332a98f92-d5cf433249bc4c309c56c9d2e02efebe on tablet d5cf433249bc4c309c56c9d2e02efebe with peer 27be1297e75b4e20ba297e7332a98f92
I20240502 21:35:40.636178 27817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1331eade23b14aee86ceeb067c93a9d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:40.649933 27817 tablet_copy_client.cc:783] T 1331eade23b14aee86ceeb067c93a9d0 P c609c27f61b9436d95dc53deaa21a3b0: tablet copy: Starting download of 0 data blocks...
I20240502 21:35:40.650427 27817 tablet_copy_client.cc:647] T 1331eade23b14aee86ceeb067c93a9d0 P c609c27f61b9436d95dc53deaa21a3b0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:35:40.653388 27817 tablet_copy_client.cc:523] T 1331eade23b14aee86ceeb067c93a9d0 P c609c27f61b9436d95dc53deaa21a3b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:35:40.657052 27817 tablet_bootstrap.cc:492] T 1331eade23b14aee86ceeb067c93a9d0 P c609c27f61b9436d95dc53deaa21a3b0: Bootstrap starting.
I20240502 21:35:40.669696 27817 log.cc:824] T 1331eade23b14aee86ceeb067c93a9d0 P c609c27f61b9436d95dc53deaa21a3b0: Log is configured to *not* fsync() on all Append() calls
I20240502 21:35:40.716351 27817 tablet_bootstrap.cc:492] T 1331eade23b14aee86ceeb067c93a9d0 P c609c27f61b9436d95dc53deaa21a3b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:35:40.716907 27817 tablet_bootstrap.cc:492] T 1331eade23b14aee86ceeb067c93a9d0 P c609c27f61b9436d95dc53deaa21a3b0: Bootstrap complete.
I20240502 21:35:40.717422 27817 ts_tablet_manager.cc:1367] T 1331eade23b14aee86ceeb067c93a9d0 P c609c27f61b9436d95dc53deaa21a3b0: Time spent bootstrapping tablet: real 0.061s	user 0.054s	sys 0.005s
I20240502 21:35:40.723704 27817 raft_consensus.cc:348] T 1331eade23b14aee86ceeb067c93a9d0 P c609c27f61b9436d95dc53deaa21a3b0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: true } }
I20240502 21:35:40.724328 27817 raft_consensus.cc:729] T 1331eade23b14aee86ceeb067c93a9d0 P c609c27f61b9436d95dc53deaa21a3b0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c609c27f61b9436d95dc53deaa21a3b0, State: Initialized, Role: LEARNER
I20240502 21:35:40.724918 27817 consensus_queue.cc:260] T 1331eade23b14aee86ceeb067c93a9d0 P c609c27f61b9436d95dc53deaa21a3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: true } }
I20240502 21:35:40.726929 27817 ts_tablet_manager.cc:1398] T 1331eade23b14aee86ceeb067c93a9d0 P c609c27f61b9436d95dc53deaa21a3b0: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20240502 21:35:40.728279 26450 tablet_copy_service.cc:342] P cb27977c77564d38afe6e64c816eacd0: Request end of tablet copy session c609c27f61b9436d95dc53deaa21a3b0-1331eade23b14aee86ceeb067c93a9d0 received from {username='slave'} at 127.27.196.69:44723
I20240502 21:35:40.728441 26450 tablet_copy_service.cc:433] P cb27977c77564d38afe6e64c816eacd0: ending tablet copy session c609c27f61b9436d95dc53deaa21a3b0-1331eade23b14aee86ceeb067c93a9d0 on tablet 1331eade23b14aee86ceeb067c93a9d0 with peer c609c27f61b9436d95dc53deaa21a3b0
I20240502 21:35:40.743909 27817 ts_tablet_manager.cc:902] T c06cdd9252864256a6870171f0f4c09c P c609c27f61b9436d95dc53deaa21a3b0: Initiating tablet copy from peer cb27977c77564d38afe6e64c816eacd0 (127.27.196.65:36209)
I20240502 21:35:40.745802 27817 tablet_copy_client.cc:308] T c06cdd9252864256a6870171f0f4c09c P c609c27f61b9436d95dc53deaa21a3b0: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:36209
I20240502 21:35:40.746840 26450 tablet_copy_service.cc:140] P cb27977c77564d38afe6e64c816eacd0: Received BeginTabletCopySession request for tablet c06cdd9252864256a6870171f0f4c09c from peer c609c27f61b9436d95dc53deaa21a3b0 ({username='slave'} at 127.27.196.69:44723)
I20240502 21:35:40.747089 26450 tablet_copy_service.cc:161] P cb27977c77564d38afe6e64c816eacd0: Beginning new tablet copy session on tablet c06cdd9252864256a6870171f0f4c09c from peer c609c27f61b9436d95dc53deaa21a3b0 at {username='slave'} at 127.27.196.69:44723: session id = c609c27f61b9436d95dc53deaa21a3b0-c06cdd9252864256a6870171f0f4c09c
I20240502 21:35:40.751026 26450 tablet_copy_source_session.cc:205] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:35:40.752748 27817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c06cdd9252864256a6870171f0f4c09c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:40.759472 27817 tablet_copy_client.cc:783] T c06cdd9252864256a6870171f0f4c09c P c609c27f61b9436d95dc53deaa21a3b0: tablet copy: Starting download of 0 data blocks...
I20240502 21:35:40.759828 27817 tablet_copy_client.cc:647] T c06cdd9252864256a6870171f0f4c09c P c609c27f61b9436d95dc53deaa21a3b0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:35:40.761960 27817 tablet_copy_client.cc:523] T c06cdd9252864256a6870171f0f4c09c P c609c27f61b9436d95dc53deaa21a3b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:35:40.765525 27817 tablet_bootstrap.cc:492] T c06cdd9252864256a6870171f0f4c09c P c609c27f61b9436d95dc53deaa21a3b0: Bootstrap starting.
I20240502 21:35:40.766795 27810 ts_tablet_manager.cc:902] T a3a929c153c348ea8a0b4e7c2cf838f6 P 27be1297e75b4e20ba297e7332a98f92: Initiating tablet copy from peer 63266faf827046629e39afa765b144bc (127.27.196.67:39203)
I20240502 21:35:40.770030 27810 tablet_copy_client.cc:308] T a3a929c153c348ea8a0b4e7c2cf838f6 P 27be1297e75b4e20ba297e7332a98f92: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:39203
I20240502 21:35:40.779173 26720 tablet_copy_service.cc:140] P 63266faf827046629e39afa765b144bc: Received BeginTabletCopySession request for tablet a3a929c153c348ea8a0b4e7c2cf838f6 from peer 27be1297e75b4e20ba297e7332a98f92 ({username='slave'} at 127.27.196.68:47389)
I20240502 21:35:40.779464 26720 tablet_copy_service.cc:161] P 63266faf827046629e39afa765b144bc: Beginning new tablet copy session on tablet a3a929c153c348ea8a0b4e7c2cf838f6 from peer 27be1297e75b4e20ba297e7332a98f92 at {username='slave'} at 127.27.196.68:47389: session id = 27be1297e75b4e20ba297e7332a98f92-a3a929c153c348ea8a0b4e7c2cf838f6
I20240502 21:35:40.782212 26720 tablet_copy_source_session.cc:205] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:35:40.784260 27810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a929c153c348ea8a0b4e7c2cf838f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:40.789865 27810 tablet_copy_client.cc:783] T a3a929c153c348ea8a0b4e7c2cf838f6 P 27be1297e75b4e20ba297e7332a98f92: tablet copy: Starting download of 0 data blocks...
I20240502 21:35:40.790234 27810 tablet_copy_client.cc:647] T a3a929c153c348ea8a0b4e7c2cf838f6 P 27be1297e75b4e20ba297e7332a98f92: tablet copy: Starting download of 1 WAL segments...
I20240502 21:35:40.792622 27810 tablet_copy_client.cc:523] T a3a929c153c348ea8a0b4e7c2cf838f6 P 27be1297e75b4e20ba297e7332a98f92: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:35:40.795575 27810 tablet_bootstrap.cc:492] T a3a929c153c348ea8a0b4e7c2cf838f6 P 27be1297e75b4e20ba297e7332a98f92: Bootstrap starting.
I20240502 21:35:40.805606 27817 tablet_bootstrap.cc:492] T c06cdd9252864256a6870171f0f4c09c P c609c27f61b9436d95dc53deaa21a3b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:35:40.806073 27817 tablet_bootstrap.cc:492] T c06cdd9252864256a6870171f0f4c09c P c609c27f61b9436d95dc53deaa21a3b0: Bootstrap complete.
I20240502 21:35:40.806370 27817 ts_tablet_manager.cc:1367] T c06cdd9252864256a6870171f0f4c09c P c609c27f61b9436d95dc53deaa21a3b0: Time spent bootstrapping tablet: real 0.041s	user 0.036s	sys 0.004s
I20240502 21:35:40.807329 27817 raft_consensus.cc:348] T c06cdd9252864256a6870171f0f4c09c P c609c27f61b9436d95dc53deaa21a3b0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: true } }
I20240502 21:35:40.807722 27817 raft_consensus.cc:729] T c06cdd9252864256a6870171f0f4c09c P c609c27f61b9436d95dc53deaa21a3b0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c609c27f61b9436d95dc53deaa21a3b0, State: Initialized, Role: LEARNER
I20240502 21:35:40.807948 27817 consensus_queue.cc:260] T c06cdd9252864256a6870171f0f4c09c P c609c27f61b9436d95dc53deaa21a3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: true } }
I20240502 21:35:40.808691 27817 ts_tablet_manager.cc:1398] T c06cdd9252864256a6870171f0f4c09c P c609c27f61b9436d95dc53deaa21a3b0: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240502 21:35:40.809785 26450 tablet_copy_service.cc:342] P cb27977c77564d38afe6e64c816eacd0: Request end of tablet copy session c609c27f61b9436d95dc53deaa21a3b0-c06cdd9252864256a6870171f0f4c09c received from {username='slave'} at 127.27.196.69:44723
I20240502 21:35:40.809984 26450 tablet_copy_service.cc:433] P cb27977c77564d38afe6e64c816eacd0: ending tablet copy session c609c27f61b9436d95dc53deaa21a3b0-c06cdd9252864256a6870171f0f4c09c on tablet c06cdd9252864256a6870171f0f4c09c with peer c609c27f61b9436d95dc53deaa21a3b0
I20240502 21:35:40.815578 27872 ts_tablet_manager.cc:902] T 8747e6794c084510a5b9e1f34c15c733 P 27be1297e75b4e20ba297e7332a98f92: Initiating tablet copy from peer 5b1043ab39c04568814296d029600092 (127.27.196.66:41285)
I20240502 21:35:40.822772 27872 tablet_copy_client.cc:308] T 8747e6794c084510a5b9e1f34c15c733 P 27be1297e75b4e20ba297e7332a98f92: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:41285
I20240502 21:35:40.830431 26586 tablet_copy_service.cc:140] P 5b1043ab39c04568814296d029600092: Received BeginTabletCopySession request for tablet 8747e6794c084510a5b9e1f34c15c733 from peer 27be1297e75b4e20ba297e7332a98f92 ({username='slave'} at 127.27.196.68:41167)
I20240502 21:35:40.830771 26586 tablet_copy_service.cc:161] P 5b1043ab39c04568814296d029600092: Beginning new tablet copy session on tablet 8747e6794c084510a5b9e1f34c15c733 from peer 27be1297e75b4e20ba297e7332a98f92 at {username='slave'} at 127.27.196.68:41167: session id = 27be1297e75b4e20ba297e7332a98f92-8747e6794c084510a5b9e1f34c15c733
I20240502 21:35:40.832612 27817 ts_tablet_manager.cc:902] T fc2c1ffd7e084141a722d0a80febd74a P c609c27f61b9436d95dc53deaa21a3b0: Initiating tablet copy from peer 5b1043ab39c04568814296d029600092 (127.27.196.66:41285)
I20240502 21:35:40.835041 26586 tablet_copy_source_session.cc:205] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:35:40.836308 27817 tablet_copy_client.cc:308] T fc2c1ffd7e084141a722d0a80febd74a P c609c27f61b9436d95dc53deaa21a3b0: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:41285
I20240502 21:35:40.837713 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8747e6794c084510a5b9e1f34c15c733. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:40.843066 27810 tablet_bootstrap.cc:492] T a3a929c153c348ea8a0b4e7c2cf838f6 P 27be1297e75b4e20ba297e7332a98f92: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:35:40.843652 27810 tablet_bootstrap.cc:492] T a3a929c153c348ea8a0b4e7c2cf838f6 P 27be1297e75b4e20ba297e7332a98f92: Bootstrap complete.
I20240502 21:35:40.844010 27810 ts_tablet_manager.cc:1367] T a3a929c153c348ea8a0b4e7c2cf838f6 P 27be1297e75b4e20ba297e7332a98f92: Time spent bootstrapping tablet: real 0.049s	user 0.043s	sys 0.004s
I20240502 21:35:40.843959 26586 tablet_copy_service.cc:140] P 5b1043ab39c04568814296d029600092: Received BeginTabletCopySession request for tablet fc2c1ffd7e084141a722d0a80febd74a from peer c609c27f61b9436d95dc53deaa21a3b0 ({username='slave'} at 127.27.196.69:41483)
I20240502 21:35:40.844342 26586 tablet_copy_service.cc:161] P 5b1043ab39c04568814296d029600092: Beginning new tablet copy session on tablet fc2c1ffd7e084141a722d0a80febd74a from peer c609c27f61b9436d95dc53deaa21a3b0 at {username='slave'} at 127.27.196.69:41483: session id = c609c27f61b9436d95dc53deaa21a3b0-fc2c1ffd7e084141a722d0a80febd74a
I20240502 21:35:40.845268 27810 raft_consensus.cc:348] T a3a929c153c348ea8a0b4e7c2cf838f6 P 27be1297e75b4e20ba297e7332a98f92 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } }
I20240502 21:35:40.845593 27810 raft_consensus.cc:729] T a3a929c153c348ea8a0b4e7c2cf838f6 P 27be1297e75b4e20ba297e7332a98f92 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 27be1297e75b4e20ba297e7332a98f92, State: Initialized, Role: LEARNER
I20240502 21:35:40.845841 27810 consensus_queue.cc:260] T a3a929c153c348ea8a0b4e7c2cf838f6 P 27be1297e75b4e20ba297e7332a98f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } }
I20240502 21:35:40.846860 27810 ts_tablet_manager.cc:1398] T a3a929c153c348ea8a0b4e7c2cf838f6 P 27be1297e75b4e20ba297e7332a98f92: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 21:35:40.846920 26586 tablet_copy_source_session.cc:205] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:35:40.847910 26720 tablet_copy_service.cc:342] P 63266faf827046629e39afa765b144bc: Request end of tablet copy session 27be1297e75b4e20ba297e7332a98f92-a3a929c153c348ea8a0b4e7c2cf838f6 received from {username='slave'} at 127.27.196.68:47389
I20240502 21:35:40.848143 26720 tablet_copy_service.cc:433] P 63266faf827046629e39afa765b144bc: ending tablet copy session 27be1297e75b4e20ba297e7332a98f92-a3a929c153c348ea8a0b4e7c2cf838f6 on tablet a3a929c153c348ea8a0b4e7c2cf838f6 with peer 27be1297e75b4e20ba297e7332a98f92
I20240502 21:35:40.848696 27817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2c1ffd7e084141a722d0a80febd74a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:40.849000 27881 ts_tablet_manager.cc:902] T 140f7280768e4ad592309e9b7d6a26da P 27be1297e75b4e20ba297e7332a98f92: Initiating tablet copy from peer cb27977c77564d38afe6e64c816eacd0 (127.27.196.65:36209)
I20240502 21:35:40.850972 27872 tablet_copy_client.cc:783] T 8747e6794c084510a5b9e1f34c15c733 P 27be1297e75b4e20ba297e7332a98f92: tablet copy: Starting download of 0 data blocks...
I20240502 21:35:40.851334 27872 tablet_copy_client.cc:647] T 8747e6794c084510a5b9e1f34c15c733 P 27be1297e75b4e20ba297e7332a98f92: tablet copy: Starting download of 1 WAL segments...
I20240502 21:35:40.854369 27872 tablet_copy_client.cc:523] T 8747e6794c084510a5b9e1f34c15c733 P 27be1297e75b4e20ba297e7332a98f92: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:35:40.855868 27817 tablet_copy_client.cc:783] T fc2c1ffd7e084141a722d0a80febd74a P c609c27f61b9436d95dc53deaa21a3b0: tablet copy: Starting download of 0 data blocks...
I20240502 21:35:40.856070 27817 tablet_copy_client.cc:647] T fc2c1ffd7e084141a722d0a80febd74a P c609c27f61b9436d95dc53deaa21a3b0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:35:40.857765 27817 tablet_copy_client.cc:523] T fc2c1ffd7e084141a722d0a80febd74a P c609c27f61b9436d95dc53deaa21a3b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:35:40.859299 27881 tablet_copy_client.cc:308] T 140f7280768e4ad592309e9b7d6a26da P 27be1297e75b4e20ba297e7332a98f92: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:36209
I20240502 21:35:40.860190 26450 tablet_copy_service.cc:140] P cb27977c77564d38afe6e64c816eacd0: Received BeginTabletCopySession request for tablet 140f7280768e4ad592309e9b7d6a26da from peer 27be1297e75b4e20ba297e7332a98f92 ({username='slave'} at 127.27.196.68:48789)
I20240502 21:35:40.860410 26450 tablet_copy_service.cc:161] P cb27977c77564d38afe6e64c816eacd0: Beginning new tablet copy session on tablet 140f7280768e4ad592309e9b7d6a26da from peer 27be1297e75b4e20ba297e7332a98f92 at {username='slave'} at 127.27.196.68:48789: session id = 27be1297e75b4e20ba297e7332a98f92-140f7280768e4ad592309e9b7d6a26da
I20240502 21:35:40.860733 27872 tablet_bootstrap.cc:492] T 8747e6794c084510a5b9e1f34c15c733 P 27be1297e75b4e20ba297e7332a98f92: Bootstrap starting.
I20240502 21:35:40.860749 27817 tablet_bootstrap.cc:492] T fc2c1ffd7e084141a722d0a80febd74a P c609c27f61b9436d95dc53deaa21a3b0: Bootstrap starting.
I20240502 21:35:40.863667 26450 tablet_copy_source_session.cc:205] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:35:40.865259 27881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 140f7280768e4ad592309e9b7d6a26da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:40.873662 27881 tablet_copy_client.cc:783] T 140f7280768e4ad592309e9b7d6a26da P 27be1297e75b4e20ba297e7332a98f92: tablet copy: Starting download of 0 data blocks...
I20240502 21:35:40.874007 27881 tablet_copy_client.cc:647] T 140f7280768e4ad592309e9b7d6a26da P 27be1297e75b4e20ba297e7332a98f92: tablet copy: Starting download of 1 WAL segments...
I20240502 21:35:40.876433 27881 tablet_copy_client.cc:523] T 140f7280768e4ad592309e9b7d6a26da P 27be1297e75b4e20ba297e7332a98f92: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:35:40.880442 27881 tablet_bootstrap.cc:492] T 140f7280768e4ad592309e9b7d6a26da P 27be1297e75b4e20ba297e7332a98f92: Bootstrap starting.
I20240502 21:35:40.903496 27872 tablet_bootstrap.cc:492] T 8747e6794c084510a5b9e1f34c15c733 P 27be1297e75b4e20ba297e7332a98f92: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:35:40.904081 27872 tablet_bootstrap.cc:492] T 8747e6794c084510a5b9e1f34c15c733 P 27be1297e75b4e20ba297e7332a98f92: Bootstrap complete.
I20240502 21:35:40.904422 27872 ts_tablet_manager.cc:1367] T 8747e6794c084510a5b9e1f34c15c733 P 27be1297e75b4e20ba297e7332a98f92: Time spent bootstrapping tablet: real 0.044s	user 0.033s	sys 0.011s
I20240502 21:35:40.905587 27872 raft_consensus.cc:348] T 8747e6794c084510a5b9e1f34c15c733 P 27be1297e75b4e20ba297e7332a98f92 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } }
I20240502 21:35:40.905902 27872 raft_consensus.cc:729] T 8747e6794c084510a5b9e1f34c15c733 P 27be1297e75b4e20ba297e7332a98f92 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 27be1297e75b4e20ba297e7332a98f92, State: Initialized, Role: LEARNER
I20240502 21:35:40.906176 27872 consensus_queue.cc:260] T 8747e6794c084510a5b9e1f34c15c733 P 27be1297e75b4e20ba297e7332a98f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } }
I20240502 21:35:40.906864 27872 ts_tablet_manager.cc:1398] T 8747e6794c084510a5b9e1f34c15c733 P 27be1297e75b4e20ba297e7332a98f92: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 21:35:40.908638 26586 tablet_copy_service.cc:342] P 5b1043ab39c04568814296d029600092: Request end of tablet copy session 27be1297e75b4e20ba297e7332a98f92-8747e6794c084510a5b9e1f34c15c733 received from {username='slave'} at 127.27.196.68:41167
I20240502 21:35:40.908960 26586 tablet_copy_service.cc:433] P 5b1043ab39c04568814296d029600092: ending tablet copy session 27be1297e75b4e20ba297e7332a98f92-8747e6794c084510a5b9e1f34c15c733 on tablet 8747e6794c084510a5b9e1f34c15c733 with peer 27be1297e75b4e20ba297e7332a98f92
I20240502 21:35:40.913242 27817 tablet_bootstrap.cc:492] T fc2c1ffd7e084141a722d0a80febd74a P c609c27f61b9436d95dc53deaa21a3b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:35:40.913743 27817 tablet_bootstrap.cc:492] T fc2c1ffd7e084141a722d0a80febd74a P c609c27f61b9436d95dc53deaa21a3b0: Bootstrap complete.
I20240502 21:35:40.914009 27817 ts_tablet_manager.cc:1367] T fc2c1ffd7e084141a722d0a80febd74a P c609c27f61b9436d95dc53deaa21a3b0: Time spent bootstrapping tablet: real 0.053s	user 0.042s	sys 0.008s
I20240502 21:35:40.914770 27817 raft_consensus.cc:348] T fc2c1ffd7e084141a722d0a80febd74a P c609c27f61b9436d95dc53deaa21a3b0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: true } }
I20240502 21:35:40.915040 27817 raft_consensus.cc:729] T fc2c1ffd7e084141a722d0a80febd74a P c609c27f61b9436d95dc53deaa21a3b0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c609c27f61b9436d95dc53deaa21a3b0, State: Initialized, Role: LEARNER
I20240502 21:35:40.915288 27817 consensus_queue.cc:260] T fc2c1ffd7e084141a722d0a80febd74a P c609c27f61b9436d95dc53deaa21a3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: true } }
I20240502 21:35:40.915803 27817 ts_tablet_manager.cc:1398] T fc2c1ffd7e084141a722d0a80febd74a P c609c27f61b9436d95dc53deaa21a3b0: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.004s
I20240502 21:35:40.916639 26586 tablet_copy_service.cc:342] P 5b1043ab39c04568814296d029600092: Request end of tablet copy session c609c27f61b9436d95dc53deaa21a3b0-fc2c1ffd7e084141a722d0a80febd74a received from {username='slave'} at 127.27.196.69:41483
I20240502 21:35:40.916780 26586 tablet_copy_service.cc:433] P 5b1043ab39c04568814296d029600092: ending tablet copy session c609c27f61b9436d95dc53deaa21a3b0-fc2c1ffd7e084141a722d0a80febd74a on tablet fc2c1ffd7e084141a722d0a80febd74a with peer c609c27f61b9436d95dc53deaa21a3b0
I20240502 21:35:40.926757 27881 tablet_bootstrap.cc:492] T 140f7280768e4ad592309e9b7d6a26da P 27be1297e75b4e20ba297e7332a98f92: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:35:40.927368 27881 tablet_bootstrap.cc:492] T 140f7280768e4ad592309e9b7d6a26da P 27be1297e75b4e20ba297e7332a98f92: Bootstrap complete.
I20240502 21:35:40.927691 27881 ts_tablet_manager.cc:1367] T 140f7280768e4ad592309e9b7d6a26da P 27be1297e75b4e20ba297e7332a98f92: Time spent bootstrapping tablet: real 0.047s	user 0.045s	sys 0.000s
I20240502 21:35:40.928570 27881 raft_consensus.cc:348] T 140f7280768e4ad592309e9b7d6a26da P 27be1297e75b4e20ba297e7332a98f92 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } }
I20240502 21:35:40.928844 27881 raft_consensus.cc:729] T 140f7280768e4ad592309e9b7d6a26da P 27be1297e75b4e20ba297e7332a98f92 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 27be1297e75b4e20ba297e7332a98f92, State: Initialized, Role: LEARNER
I20240502 21:35:40.929075 27881 consensus_queue.cc:260] T 140f7280768e4ad592309e9b7d6a26da P 27be1297e75b4e20ba297e7332a98f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } }
I20240502 21:35:40.929643 27881 ts_tablet_manager.cc:1398] T 140f7280768e4ad592309e9b7d6a26da P 27be1297e75b4e20ba297e7332a98f92: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 21:35:40.930756 26450 tablet_copy_service.cc:342] P cb27977c77564d38afe6e64c816eacd0: Request end of tablet copy session 27be1297e75b4e20ba297e7332a98f92-140f7280768e4ad592309e9b7d6a26da received from {username='slave'} at 127.27.196.68:48789
I20240502 21:35:40.930949 26450 tablet_copy_service.cc:433] P cb27977c77564d38afe6e64c816eacd0: ending tablet copy session 27be1297e75b4e20ba297e7332a98f92-140f7280768e4ad592309e9b7d6a26da on tablet 140f7280768e4ad592309e9b7d6a26da with peer 27be1297e75b4e20ba297e7332a98f92
I20240502 21:35:41.143968 27375 raft_consensus.cc:1203] T c06cdd9252864256a6870171f0f4c09c P c609c27f61b9436d95dc53deaa21a3b0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:35:41.156263 27510 raft_consensus.cc:1203] T d5cf433249bc4c309c56c9d2e02efebe P 27be1297e75b4e20ba297e7332a98f92 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:35:41.203655 27375 raft_consensus.cc:1203] T 1331eade23b14aee86ceeb067c93a9d0 P c609c27f61b9436d95dc53deaa21a3b0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:35:41.206250 27510 raft_consensus.cc:1203] T 8747e6794c084510a5b9e1f34c15c733 P 27be1297e75b4e20ba297e7332a98f92 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:35:41.312462 27375 raft_consensus.cc:1203] T fc2c1ffd7e084141a722d0a80febd74a P c609c27f61b9436d95dc53deaa21a3b0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:35:41.388904 27510 raft_consensus.cc:1203] T a3a929c153c348ea8a0b4e7c2cf838f6 P 27be1297e75b4e20ba297e7332a98f92 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:35:41.428349 27510 raft_consensus.cc:1203] T 140f7280768e4ad592309e9b7d6a26da P 27be1297e75b4e20ba297e7332a98f92 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:35:41.607069 27925 raft_consensus.cc:1050] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0: attempting to promote NON_VOTER 27be1297e75b4e20ba297e7332a98f92 to VOTER
I20240502 21:35:41.608418 27925 consensus_queue.cc:237] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } }
I20240502 21:35:41.611570 27510 raft_consensus.cc:1261] T d5cf433249bc4c309c56c9d2e02efebe P 27be1297e75b4e20ba297e7332a98f92 [term 1 LEARNER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:35:41.611712 26697 raft_consensus.cc:1261] T d5cf433249bc4c309c56c9d2e02efebe P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:35:41.611887 26564 raft_consensus.cc:1261] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:35:41.612628 27711 consensus_queue.cc:1035] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:35:41.613004 27653 consensus_queue.cc:1035] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:35:41.616690 28004 consensus_queue.cc:1035] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:35:41.618357 27711 raft_consensus.cc:2932] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:41.619064 26697 raft_consensus.cc:2932] T d5cf433249bc4c309c56c9d2e02efebe P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:41.622573 26564 raft_consensus.cc:2932] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:41.623862 26284 catalog_manager.cc:5484] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 reported cstate change: config changed from index 4 to 5, 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cb27977c77564d38afe6e64c816eacd0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:35:41.624014 27510 raft_consensus.cc:2932] T d5cf433249bc4c309c56c9d2e02efebe P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:41.639789 27759 raft_consensus.cc:1050] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092: attempting to promote NON_VOTER 27be1297e75b4e20ba297e7332a98f92 to VOTER
I20240502 21:35:41.640730 27759 consensus_queue.cc:237] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } }
I20240502 21:35:41.643680 26697 raft_consensus.cc:1261] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Refusing update from remote peer 5b1043ab39c04568814296d029600092: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:35:41.644369 27510 raft_consensus.cc:1261] T 8747e6794c084510a5b9e1f34c15c733 P 27be1297e75b4e20ba297e7332a98f92 [term 1 LEARNER]: Refusing update from remote peer 5b1043ab39c04568814296d029600092: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:35:41.644459 26429 raft_consensus.cc:1261] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Refusing update from remote peer 5b1043ab39c04568814296d029600092: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:35:41.644934 27759 consensus_queue.cc:1035] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:35:41.645298 27953 consensus_queue.cc:1035] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:35:41.646152 28013 consensus_queue.cc:1035] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:35:41.661062 27926 raft_consensus.cc:2932] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:41.662647 27510 raft_consensus.cc:2932] T 8747e6794c084510a5b9e1f34c15c733 P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:41.668141 26697 raft_consensus.cc:2932] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:41.669268 26287 catalog_manager.cc:5484] T 8747e6794c084510a5b9e1f34c15c733 P 27be1297e75b4e20ba297e7332a98f92 reported cstate change: config changed from index 4 to 5, 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b1043ab39c04568814296d029600092" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:41.669219 26429 raft_consensus.cc:2932] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:41.676115 26564 consensus_queue.cc:237] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } }
I20240502 21:35:41.678690 26697 raft_consensus.cc:1261] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Refusing update from remote peer 5b1043ab39c04568814296d029600092: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:35:41.678907 27510 raft_consensus.cc:1261] T 8747e6794c084510a5b9e1f34c15c733 P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Refusing update from remote peer 5b1043ab39c04568814296d029600092: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:35:41.679543 28013 consensus_queue.cc:1035] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:35:41.680187 28013 consensus_queue.cc:1035] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:35:41.682291 28013 raft_consensus.cc:2932] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cb27977c77564d38afe6e64c816eacd0 (127.27.196.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:41.684307 27510 raft_consensus.cc:2932] T 8747e6794c084510a5b9e1f34c15c733 P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cb27977c77564d38afe6e64c816eacd0 (127.27.196.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:41.684651 26698 raft_consensus.cc:2932] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cb27977c77564d38afe6e64c816eacd0 (127.27.196.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:41.686805 26273 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8747e6794c084510a5b9e1f34c15c733 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:35:41.690213 26287 catalog_manager.cc:5484] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 reported cstate change: config changed from index 5 to 6, VOTER cb27977c77564d38afe6e64c816eacd0 (127.27.196.65) evicted. New cstate: current_term: 1 leader_uuid: "5b1043ab39c04568814296d029600092" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:35:41.695142 26409 tablet_service.cc:1514] Processing DeleteTablet for tablet 8747e6794c084510a5b9e1f34c15c733 with delete_type TABLET_DATA_TOMBSTONED (TS cb27977c77564d38afe6e64c816eacd0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33746
I20240502 21:35:41.696202 28017 tablet_replica.cc:330] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0: stopping tablet replica
I20240502 21:35:41.696707 28017 raft_consensus.cc:2229] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:35:41.697044 28017 raft_consensus.cc:2258] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:35:41.698872 28017 ts_tablet_manager.cc:1875] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:35:41.704188 28017 ts_tablet_manager.cc:1888] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:35:41.704378 28017 log.cc:1196] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/wals/8747e6794c084510a5b9e1f34c15c733
I20240502 21:35:41.705358 26273 catalog_manager.cc:4830] TS cb27977c77564d38afe6e64c816eacd0 (127.27.196.65:36209): tablet 8747e6794c084510a5b9e1f34c15c733 (table rebalance_test_table_0 [id=7277b1f7f8b747f4a8e65521bf4af6c0]) successfully deleted
I20240502 21:35:41.750377 27925 raft_consensus.cc:1050] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0: attempting to promote NON_VOTER c609c27f61b9436d95dc53deaa21a3b0 to VOTER
I20240502 21:35:41.751154 27925 consensus_queue.cc:237] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } }
I20240502 21:35:41.753516 26698 raft_consensus.cc:1261] T c06cdd9252864256a6870171f0f4c09c P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:35:41.753644 26564 raft_consensus.cc:1261] T c06cdd9252864256a6870171f0f4c09c P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:35:41.753661 27375 raft_consensus.cc:1261] T c06cdd9252864256a6870171f0f4c09c P c609c27f61b9436d95dc53deaa21a3b0 [term 1 LEARNER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:35:41.754246 27925 consensus_queue.cc:1035] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:35:41.754568 27653 consensus_queue.cc:1035] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:35:41.754884 27711 consensus_queue.cc:1035] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:35:41.758086 27711 raft_consensus.cc:2932] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:41.759439 26698 raft_consensus.cc:2932] T c06cdd9252864256a6870171f0f4c09c P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:41.760339 27375 raft_consensus.cc:2932] T c06cdd9252864256a6870171f0f4c09c P c609c27f61b9436d95dc53deaa21a3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:41.764535 26564 raft_consensus.cc:2932] T c06cdd9252864256a6870171f0f4c09c P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:41.766875 27653 raft_consensus.cc:1050] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0: attempting to promote NON_VOTER c609c27f61b9436d95dc53deaa21a3b0 to VOTER
I20240502 21:35:41.767939 27653 consensus_queue.cc:237] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } }
I20240502 21:35:41.769312 26282 catalog_manager.cc:5484] T c06cdd9252864256a6870171f0f4c09c P c609c27f61b9436d95dc53deaa21a3b0 reported cstate change: config changed from index 4 to 5, c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cb27977c77564d38afe6e64c816eacd0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:41.772890 26698 raft_consensus.cc:1261] T 1331eade23b14aee86ceeb067c93a9d0 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:35:41.773653 27375 raft_consensus.cc:1261] T 1331eade23b14aee86ceeb067c93a9d0 P c609c27f61b9436d95dc53deaa21a3b0 [term 1 LEARNER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:35:41.773833 27711 consensus_queue.cc:1035] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:35:41.774216 26564 raft_consensus.cc:1261] T 1331eade23b14aee86ceeb067c93a9d0 P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:35:41.774412 27653 consensus_queue.cc:1035] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:35:41.775458 27925 consensus_queue.cc:1035] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:35:41.776559 26429 consensus_queue.cc:237] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } }
I20240502 21:35:41.779557 27374 raft_consensus.cc:1261] T c06cdd9252864256a6870171f0f4c09c P c609c27f61b9436d95dc53deaa21a3b0 [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:35:41.780400 26697 raft_consensus.cc:1261] T c06cdd9252864256a6870171f0f4c09c P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:35:41.780510 27711 consensus_queue.cc:1035] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:35:41.781077 27925 consensus_queue.cc:1035] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:35:41.784801 28004 raft_consensus.cc:2932] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5b1043ab39c04568814296d029600092 (127.27.196.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:41.785626 27953 raft_consensus.cc:1050] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092: attempting to promote NON_VOTER c609c27f61b9436d95dc53deaa21a3b0 to VOTER
I20240502 21:35:41.786592 27953 consensus_queue.cc:237] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } }
I20240502 21:35:41.787806 27375 raft_consensus.cc:2932] T c06cdd9252864256a6870171f0f4c09c P c609c27f61b9436d95dc53deaa21a3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5b1043ab39c04568814296d029600092 (127.27.196.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:41.788780 26697 raft_consensus.cc:2932] T c06cdd9252864256a6870171f0f4c09c P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5b1043ab39c04568814296d029600092 (127.27.196.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:41.791534 26273 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c06cdd9252864256a6870171f0f4c09c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:35:41.791251 26282 catalog_manager.cc:5484] T c06cdd9252864256a6870171f0f4c09c P cb27977c77564d38afe6e64c816eacd0 reported cstate change: config changed from index 5 to 6, VOTER 5b1043ab39c04568814296d029600092 (127.27.196.66) evicted. New cstate: current_term: 1 leader_uuid: "cb27977c77564d38afe6e64c816eacd0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:35:41.795245 27375 raft_consensus.cc:1261] T fc2c1ffd7e084141a722d0a80febd74a P c609c27f61b9436d95dc53deaa21a3b0 [term 1 LEARNER]: Refusing update from remote peer 5b1043ab39c04568814296d029600092: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:35:41.795476 26429 raft_consensus.cc:1261] T fc2c1ffd7e084141a722d0a80febd74a P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Refusing update from remote peer 5b1043ab39c04568814296d029600092: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:35:41.795089 26565 raft_consensus.cc:2932] T 1331eade23b14aee86ceeb067c93a9d0 P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:41.795802 27926 consensus_queue.cc:1035] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:35:41.796504 27926 consensus_queue.cc:1035] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:35:41.799615 26696 raft_consensus.cc:1261] T fc2c1ffd7e084141a722d0a80febd74a P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Refusing update from remote peer 5b1043ab39c04568814296d029600092: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:35:41.800285 26696 raft_consensus.cc:2932] T 1331eade23b14aee86ceeb067c93a9d0 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:41.800681 27926 consensus_queue.cc:1035] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:35:41.801373 26287 catalog_manager.cc:5484] T 1331eade23b14aee86ceeb067c93a9d0 P 5b1043ab39c04568814296d029600092 reported cstate change: config changed from index 4 to 5, c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cb27977c77564d38afe6e64c816eacd0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:41.793864 28004 raft_consensus.cc:2932] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:41.803927 26543 tablet_service.cc:1514] Processing DeleteTablet for tablet c06cdd9252864256a6870171f0f4c09c with delete_type TABLET_DATA_TOMBSTONED (TS 5b1043ab39c04568814296d029600092 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34904
I20240502 21:35:41.808311 27374 raft_consensus.cc:2932] T 1331eade23b14aee86ceeb067c93a9d0 P c609c27f61b9436d95dc53deaa21a3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:41.821210 27953 raft_consensus.cc:2932] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:41.822232 26429 raft_consensus.cc:2932] T fc2c1ffd7e084141a722d0a80febd74a P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:41.826484 26430 consensus_queue.cc:237] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } }
I20240502 21:35:41.828704 27374 raft_consensus.cc:1261] T 1331eade23b14aee86ceeb067c93a9d0 P c609c27f61b9436d95dc53deaa21a3b0 [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:35:41.829351 26565 raft_consensus.cc:1261] T 1331eade23b14aee86ceeb067c93a9d0 P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:35:41.829860 26286 catalog_manager.cc:5484] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 reported cstate change: config changed from index 4 to 5, c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b1043ab39c04568814296d029600092" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:35:41.830171 28004 consensus_queue.cc:1035] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:35:41.830870 28004 consensus_queue.cc:1035] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:35:41.831205 27374 raft_consensus.cc:2932] T fc2c1ffd7e084141a722d0a80febd74a P c609c27f61b9436d95dc53deaa21a3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:41.835081 27925 raft_consensus.cc:2932] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63266faf827046629e39afa765b144bc (127.27.196.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:41.836586 26697 raft_consensus.cc:2932] T fc2c1ffd7e084141a722d0a80febd74a P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:41.840111 26273 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1331eade23b14aee86ceeb067c93a9d0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:35:41.839881 27375 raft_consensus.cc:2932] T 1331eade23b14aee86ceeb067c93a9d0 P c609c27f61b9436d95dc53deaa21a3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63266faf827046629e39afa765b144bc (127.27.196.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:41.842411 26286 catalog_manager.cc:5484] T 1331eade23b14aee86ceeb067c93a9d0 P cb27977c77564d38afe6e64c816eacd0 reported cstate change: config changed from index 5 to 6, VOTER 63266faf827046629e39afa765b144bc (127.27.196.67) evicted. New cstate: current_term: 1 leader_uuid: "cb27977c77564d38afe6e64c816eacd0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:35:41.845599 26564 consensus_queue.cc:237] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } }
I20240502 21:35:41.849263 26697 raft_consensus.cc:1261] T fc2c1ffd7e084141a722d0a80febd74a P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Refusing update from remote peer 5b1043ab39c04568814296d029600092: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:35:41.850114 28013 consensus_queue.cc:1035] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:35:41.850236 26678 tablet_service.cc:1514] Processing DeleteTablet for tablet 1331eade23b14aee86ceeb067c93a9d0 with delete_type TABLET_DATA_TOMBSTONED (TS 63266faf827046629e39afa765b144bc not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42438
I20240502 21:35:41.850611 27374 raft_consensus.cc:1261] T fc2c1ffd7e084141a722d0a80febd74a P c609c27f61b9436d95dc53deaa21a3b0 [term 1 FOLLOWER]: Refusing update from remote peer 5b1043ab39c04568814296d029600092: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:35:41.851172 28013 consensus_queue.cc:1035] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:35:41.852481 26565 raft_consensus.cc:2932] T 1331eade23b14aee86ceeb067c93a9d0 P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63266faf827046629e39afa765b144bc (127.27.196.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:41.854125 28013 raft_consensus.cc:2932] T fc2c1ffd7e084141a722d0a80febd74a P 5b1043ab39c04568814296d029600092 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cb27977c77564d38afe6e64c816eacd0 (127.27.196.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:41.859051 26697 raft_consensus.cc:2932] T fc2c1ffd7e084141a722d0a80febd74a P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cb27977c77564d38afe6e64c816eacd0 (127.27.196.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:41.856360 27373 raft_consensus.cc:2932] T fc2c1ffd7e084141a722d0a80febd74a P c609c27f61b9436d95dc53deaa21a3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cb27977c77564d38afe6e64c816eacd0 (127.27.196.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:41.863554 28029 tablet_replica.cc:330] T c06cdd9252864256a6870171f0f4c09c P 5b1043ab39c04568814296d029600092: stopping tablet replica
I20240502 21:35:41.864037 28029 raft_consensus.cc:2229] T c06cdd9252864256a6870171f0f4c09c P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:35:41.864483 28029 raft_consensus.cc:2258] T c06cdd9252864256a6870171f0f4c09c P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:35:41.866451 28029 ts_tablet_manager.cc:1875] T c06cdd9252864256a6870171f0f4c09c P 5b1043ab39c04568814296d029600092: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:35:41.867708 26273 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fc2c1ffd7e084141a722d0a80febd74a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:35:41.869366 26282 catalog_manager.cc:5484] T fc2c1ffd7e084141a722d0a80febd74a P c609c27f61b9436d95dc53deaa21a3b0 reported cstate change: config changed from index 5 to 6, VOTER cb27977c77564d38afe6e64c816eacd0 (127.27.196.65) evicted. New cstate: current_term: 1 leader_uuid: "5b1043ab39c04568814296d029600092" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:41.871773 28041 tablet_replica.cc:330] T 1331eade23b14aee86ceeb067c93a9d0 P 63266faf827046629e39afa765b144bc: stopping tablet replica
I20240502 21:35:41.872328 28041 raft_consensus.cc:2229] T 1331eade23b14aee86ceeb067c93a9d0 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:35:41.872956 28041 raft_consensus.cc:2258] T 1331eade23b14aee86ceeb067c93a9d0 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:35:41.874660 26409 tablet_service.cc:1514] Processing DeleteTablet for tablet fc2c1ffd7e084141a722d0a80febd74a with delete_type TABLET_DATA_TOMBSTONED (TS cb27977c77564d38afe6e64c816eacd0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33746
I20240502 21:35:41.875257 28017 tablet_replica.cc:330] T fc2c1ffd7e084141a722d0a80febd74a P cb27977c77564d38afe6e64c816eacd0: stopping tablet replica
I20240502 21:35:41.875532 28017 raft_consensus.cc:2229] T fc2c1ffd7e084141a722d0a80febd74a P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:35:41.875805 28041 ts_tablet_manager.cc:1875] T 1331eade23b14aee86ceeb067c93a9d0 P 63266faf827046629e39afa765b144bc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:35:41.876132 28017 raft_consensus.cc:2258] T fc2c1ffd7e084141a722d0a80febd74a P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:35:41.877338 28017 ts_tablet_manager.cc:1875] T fc2c1ffd7e084141a722d0a80febd74a P cb27977c77564d38afe6e64c816eacd0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:35:41.884373 28041 ts_tablet_manager.cc:1888] T 1331eade23b14aee86ceeb067c93a9d0 P 63266faf827046629e39afa765b144bc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:35:41.884732 28041 log.cc:1196] T 1331eade23b14aee86ceeb067c93a9d0 P 63266faf827046629e39afa765b144bc: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/wals/1331eade23b14aee86ceeb067c93a9d0
I20240502 21:35:41.884966 28017 ts_tablet_manager.cc:1888] T fc2c1ffd7e084141a722d0a80febd74a P cb27977c77564d38afe6e64c816eacd0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:35:41.885293 28017 log.cc:1196] T fc2c1ffd7e084141a722d0a80febd74a P cb27977c77564d38afe6e64c816eacd0: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/wals/fc2c1ffd7e084141a722d0a80febd74a
I20240502 21:35:41.885937 28029 ts_tablet_manager.cc:1888] T c06cdd9252864256a6870171f0f4c09c P 5b1043ab39c04568814296d029600092: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:35:41.886231 28029 log.cc:1196] T c06cdd9252864256a6870171f0f4c09c P 5b1043ab39c04568814296d029600092: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/wals/c06cdd9252864256a6870171f0f4c09c
I20240502 21:35:41.886359 26273 catalog_manager.cc:4830] TS cb27977c77564d38afe6e64c816eacd0 (127.27.196.65:36209): tablet fc2c1ffd7e084141a722d0a80febd74a (table rebalance_test_table_2 [id=79000138ee6c4b96965a29a347143ffa]) successfully deleted
I20240502 21:35:41.886445 26274 catalog_manager.cc:4830] TS 63266faf827046629e39afa765b144bc (127.27.196.67:39203): tablet 1331eade23b14aee86ceeb067c93a9d0 (table rebalance_test_table_1 [id=cc5055b898c74d0ea2d82c8d8f1f5bb4]) successfully deleted
I20240502 21:35:41.887396 26273 catalog_manager.cc:4830] TS 5b1043ab39c04568814296d029600092 (127.27.196.66:41285): tablet c06cdd9252864256a6870171f0f4c09c (table rebalance_test_table_0 [id=7277b1f7f8b747f4a8e65521bf4af6c0]) successfully deleted
I20240502 21:35:41.917711 27711 raft_consensus.cc:1050] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0: attempting to promote NON_VOTER 27be1297e75b4e20ba297e7332a98f92 to VOTER
I20240502 21:35:41.918534 27711 consensus_queue.cc:237] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } }
I20240502 21:35:41.922021 26697 raft_consensus.cc:1261] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:35:41.922784 27510 raft_consensus.cc:1261] T 140f7280768e4ad592309e9b7d6a26da P 27be1297e75b4e20ba297e7332a98f92 [term 1 LEARNER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:35:41.922901 26565 raft_consensus.cc:1261] T 140f7280768e4ad592309e9b7d6a26da P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:35:41.922932 27711 consensus_queue.cc:1035] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:35:41.923592 28004 consensus_queue.cc:1035] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:35:41.924250 27925 consensus_queue.cc:1035] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:35:41.930044 27711 raft_consensus.cc:2932] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:41.932520 26565 raft_consensus.cc:2932] T 140f7280768e4ad592309e9b7d6a26da P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:41.935057 26697 raft_consensus.cc:2932] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:41.937052 27510 raft_consensus.cc:2932] T 140f7280768e4ad592309e9b7d6a26da P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:41.940768 26286 catalog_manager.cc:5484] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 reported cstate change: config changed from index 4 to 5, 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cb27977c77564d38afe6e64c816eacd0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:35:41.946735 26430 consensus_queue.cc:237] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } }
I20240502 21:35:41.949446 27510 raft_consensus.cc:1261] T 140f7280768e4ad592309e9b7d6a26da P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:35:41.949865 26565 raft_consensus.cc:1261] T 140f7280768e4ad592309e9b7d6a26da P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:35:41.950963 27653 consensus_queue.cc:1035] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:35:41.951356 27711 consensus_queue.cc:1035] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:35:41.953820 28004 raft_consensus.cc:2932] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63266faf827046629e39afa765b144bc (127.27.196.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:41.954775 27510 raft_consensus.cc:2932] T 140f7280768e4ad592309e9b7d6a26da P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63266faf827046629e39afa765b144bc (127.27.196.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:41.957939 26273 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 140f7280768e4ad592309e9b7d6a26da with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:35:41.958182 26565 raft_consensus.cc:2932] T 140f7280768e4ad592309e9b7d6a26da P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63266faf827046629e39afa765b144bc (127.27.196.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:41.959781 26282 catalog_manager.cc:5484] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 reported cstate change: config changed from index 5 to 6, VOTER 63266faf827046629e39afa765b144bc (127.27.196.67) evicted. New cstate: current_term: 1 leader_uuid: "cb27977c77564d38afe6e64c816eacd0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:35:41.963899 26678 tablet_service.cc:1514] Processing DeleteTablet for tablet 140f7280768e4ad592309e9b7d6a26da with delete_type TABLET_DATA_TOMBSTONED (TS 63266faf827046629e39afa765b144bc not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42438
I20240502 21:35:41.964323 28041 tablet_replica.cc:330] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc: stopping tablet replica
I20240502 21:35:41.964581 28041 raft_consensus.cc:2229] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:35:41.964880 28041 raft_consensus.cc:2258] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:35:41.966075 28041 ts_tablet_manager.cc:1875] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:35:41.969467 28075 raft_consensus.cc:1050] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc: attempting to promote NON_VOTER 27be1297e75b4e20ba297e7332a98f92 to VOTER
I20240502 21:35:41.970711 28075 consensus_queue.cc:237] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } }
I20240502 21:35:41.972406 28041 ts_tablet_manager.cc:1888] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:35:41.972628 28041 log.cc:1196] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/wals/140f7280768e4ad592309e9b7d6a26da
I20240502 21:35:41.973628 26274 catalog_manager.cc:4830] TS 63266faf827046629e39afa765b144bc (127.27.196.67:39203): tablet 140f7280768e4ad592309e9b7d6a26da (table rebalance_test_table_1 [id=cc5055b898c74d0ea2d82c8d8f1f5bb4]) successfully deleted
I20240502 21:35:41.973845 27510 raft_consensus.cc:1261] T a3a929c153c348ea8a0b4e7c2cf838f6 P 27be1297e75b4e20ba297e7332a98f92 [term 1 LEARNER]: Refusing update from remote peer 63266faf827046629e39afa765b144bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:35:41.974586 27966 consensus_queue.cc:1035] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:35:41.974691 26430 raft_consensus.cc:1261] T a3a929c153c348ea8a0b4e7c2cf838f6 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Refusing update from remote peer 63266faf827046629e39afa765b144bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:35:41.974828 26565 raft_consensus.cc:1261] T a3a929c153c348ea8a0b4e7c2cf838f6 P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Refusing update from remote peer 63266faf827046629e39afa765b144bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:35:41.975363 27966 consensus_queue.cc:1035] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:35:41.975770 28075 consensus_queue.cc:1035] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:35:41.979418 27966 raft_consensus.cc:2932] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:41.979673 26430 raft_consensus.cc:2932] T a3a929c153c348ea8a0b4e7c2cf838f6 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:41.985843 26284 catalog_manager.cc:5484] T a3a929c153c348ea8a0b4e7c2cf838f6 P cb27977c77564d38afe6e64c816eacd0 reported cstate change: config changed from index 4 to 5, 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63266faf827046629e39afa765b144bc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:41.988955 27510 raft_consensus.cc:2932] T a3a929c153c348ea8a0b4e7c2cf838f6 P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:41.999445 26565 raft_consensus.cc:2932] T a3a929c153c348ea8a0b4e7c2cf838f6 P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:42.003055 26697 consensus_queue.cc:237] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } }
I20240502 21:35:42.005846 26565 raft_consensus.cc:1261] T a3a929c153c348ea8a0b4e7c2cf838f6 P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Refusing update from remote peer 63266faf827046629e39afa765b144bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:35:42.005860 27510 raft_consensus.cc:1261] T a3a929c153c348ea8a0b4e7c2cf838f6 P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Refusing update from remote peer 63266faf827046629e39afa765b144bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:35:42.006501 28081 consensus_queue.cc:1035] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:35:42.006753 28075 consensus_queue.cc:1035] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:35:42.009656 27966 raft_consensus.cc:2932] T a3a929c153c348ea8a0b4e7c2cf838f6 P 63266faf827046629e39afa765b144bc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cb27977c77564d38afe6e64c816eacd0 (127.27.196.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:42.010416 27510 raft_consensus.cc:2932] T a3a929c153c348ea8a0b4e7c2cf838f6 P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cb27977c77564d38afe6e64c816eacd0 (127.27.196.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:42.014648 26274 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a3a929c153c348ea8a0b4e7c2cf838f6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:35:42.016191 26282 catalog_manager.cc:5484] T a3a929c153c348ea8a0b4e7c2cf838f6 P 27be1297e75b4e20ba297e7332a98f92 reported cstate change: config changed from index 5 to 6, VOTER cb27977c77564d38afe6e64c816eacd0 (127.27.196.65) evicted. New cstate: current_term: 1 leader_uuid: "63266faf827046629e39afa765b144bc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:42.016850 26565 raft_consensus.cc:2932] T a3a929c153c348ea8a0b4e7c2cf838f6 P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cb27977c77564d38afe6e64c816eacd0 (127.27.196.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:42.021410 26409 tablet_service.cc:1514] Processing DeleteTablet for tablet a3a929c153c348ea8a0b4e7c2cf838f6 with delete_type TABLET_DATA_TOMBSTONED (TS cb27977c77564d38afe6e64c816eacd0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33746
I20240502 21:35:42.021813 28017 tablet_replica.cc:330] T a3a929c153c348ea8a0b4e7c2cf838f6 P cb27977c77564d38afe6e64c816eacd0: stopping tablet replica
I20240502 21:35:42.022030 28017 raft_consensus.cc:2229] T a3a929c153c348ea8a0b4e7c2cf838f6 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:35:42.022374 28017 raft_consensus.cc:2258] T a3a929c153c348ea8a0b4e7c2cf838f6 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:35:42.023545 28017 ts_tablet_manager.cc:1875] T a3a929c153c348ea8a0b4e7c2cf838f6 P cb27977c77564d38afe6e64c816eacd0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:35:42.029130 28017 ts_tablet_manager.cc:1888] T a3a929c153c348ea8a0b4e7c2cf838f6 P cb27977c77564d38afe6e64c816eacd0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:35:42.029297 28017 log.cc:1196] T a3a929c153c348ea8a0b4e7c2cf838f6 P cb27977c77564d38afe6e64c816eacd0: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/wals/a3a929c153c348ea8a0b4e7c2cf838f6
I20240502 21:35:42.030045 26273 catalog_manager.cc:4830] TS cb27977c77564d38afe6e64c816eacd0 (127.27.196.65:36209): tablet a3a929c153c348ea8a0b4e7c2cf838f6 (table rebalance_test_table_0 [id=7277b1f7f8b747f4a8e65521bf4af6c0]) successfully deleted
I20240502 21:35:42.119907 26430 consensus_queue.cc:237] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } }
I20240502 21:35:42.121841 27510 raft_consensus.cc:1261] T d5cf433249bc4c309c56c9d2e02efebe P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:35:42.121932 26698 raft_consensus.cc:1261] T d5cf433249bc4c309c56c9d2e02efebe P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:35:42.122493 28004 consensus_queue.cc:1035] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:35:42.123031 28004 consensus_queue.cc:1035] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:35:42.125257 27653 raft_consensus.cc:2932] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5b1043ab39c04568814296d029600092 (127.27.196.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:42.125929 27510 raft_consensus.cc:2932] T d5cf433249bc4c309c56c9d2e02efebe P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5b1043ab39c04568814296d029600092 (127.27.196.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:42.125973 26698 raft_consensus.cc:2932] T d5cf433249bc4c309c56c9d2e02efebe P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5b1043ab39c04568814296d029600092 (127.27.196.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:42.128651 26273 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d5cf433249bc4c309c56c9d2e02efebe with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:35:42.130666 26282 catalog_manager.cc:5484] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 reported cstate change: config changed from index 5 to 6, VOTER 5b1043ab39c04568814296d029600092 (127.27.196.66) evicted. New cstate: current_term: 1 leader_uuid: "cb27977c77564d38afe6e64c816eacd0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:35:42.135037 26543 tablet_service.cc:1514] Processing DeleteTablet for tablet d5cf433249bc4c309c56c9d2e02efebe with delete_type TABLET_DATA_TOMBSTONED (TS 5b1043ab39c04568814296d029600092 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34904
I20240502 21:35:42.135478 28029 tablet_replica.cc:330] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092: stopping tablet replica
I20240502 21:35:42.135654 28029 raft_consensus.cc:2229] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:35:42.135944 28029 raft_consensus.cc:2258] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:35:42.136761 28029 ts_tablet_manager.cc:1875] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:35:42.141376 28029 ts_tablet_manager.cc:1888] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:35:42.141501 28029 log.cc:1196] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/wals/d5cf433249bc4c309c56c9d2e02efebe
I20240502 21:35:42.142333 26273 catalog_manager.cc:4830] TS 5b1043ab39c04568814296d029600092 (127.27.196.66:41285): tablet d5cf433249bc4c309c56c9d2e02efebe (table rebalance_test_table_2 [id=79000138ee6c4b96965a29a347143ffa]) successfully deleted
I20240502 21:35:42.386569 26678 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:42.401363 27354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:35:42.408041 26543 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:35:42.408656 27490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:35:42.422658 26409 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:35:42.492270 26565 consensus_queue.cc:237] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: true } }
I20240502 21:35:42.494813 26430 raft_consensus.cc:1261] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Refusing update from remote peer 5b1043ab39c04568814296d029600092: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:35:42.494935 26497 consensus_peers.cc:485] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 -> Peer c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69:43269): Couldn't send request to peer c609c27f61b9436d95dc53deaa21a3b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5bca5fdd4d14b5baaa0b0cfc4754a23. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:42.495219 26698 raft_consensus.cc:1261] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Refusing update from remote peer 5b1043ab39c04568814296d029600092: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:35:42.495528 28152 consensus_queue.cc:1035] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:35:42.495844 27953 consensus_queue.cc:1035] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:35:42.499038 27953 raft_consensus.cc:2932] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: true } } }
I20240502 21:35:42.499590 26698 raft_consensus.cc:2932] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: true } } }
I20240502 21:35:42.504140 26282 catalog_manager.cc:5484] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc reported cstate change: config changed from index -1 to 4, NON_VOTER c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) added. New cstate: current_term: 1 leader_uuid: "5b1043ab39c04568814296d029600092" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: true } } }
I20240502 21:35:42.518900 26430 raft_consensus.cc:2932] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: true } } }
I20240502 21:35:42.529825 26698 consensus_queue.cc:237] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } }
I20240502 21:35:42.532244 26430 raft_consensus.cc:1261] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Refusing update from remote peer 63266faf827046629e39afa765b144bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:35:42.532610 26632 consensus_peers.cc:485] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc -> Peer 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68:35953): Couldn't send request to peer 27be1297e75b4e20ba297e7332a98f92. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 35508ded8421457288bdef00bf12f698. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:42.532912 28162 consensus_queue.cc:1035] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:35:42.532994 26565 raft_consensus.cc:1261] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Refusing update from remote peer 63266faf827046629e39afa765b144bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:35:42.533452 28162 consensus_queue.cc:1035] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:35:42.535881 28081 raft_consensus.cc:2932] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } } }
I20240502 21:35:42.536541 26430 raft_consensus.cc:2932] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } } }
I20240502 21:35:42.539644 26565 raft_consensus.cc:2932] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } } }
I20240502 21:35:42.542256 26284 catalog_manager.cc:5484] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 reported cstate change: config changed from index -1 to 4, NON_VOTER 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) added. New cstate: current_term: 1 leader_uuid: "63266faf827046629e39afa765b144bc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } } }
I20240502 21:35:42.939719 28187 ts_tablet_manager.cc:902] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P c609c27f61b9436d95dc53deaa21a3b0: Initiating tablet copy from peer 5b1043ab39c04568814296d029600092 (127.27.196.66:41285)
I20240502 21:35:42.940897 28187 tablet_copy_client.cc:308] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P c609c27f61b9436d95dc53deaa21a3b0: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:41285
I20240502 21:35:42.941757 26586 tablet_copy_service.cc:140] P 5b1043ab39c04568814296d029600092: Received BeginTabletCopySession request for tablet e5bca5fdd4d14b5baaa0b0cfc4754a23 from peer c609c27f61b9436d95dc53deaa21a3b0 ({username='slave'} at 127.27.196.69:41483)
I20240502 21:35:42.941926 26586 tablet_copy_service.cc:161] P 5b1043ab39c04568814296d029600092: Beginning new tablet copy session on tablet e5bca5fdd4d14b5baaa0b0cfc4754a23 from peer c609c27f61b9436d95dc53deaa21a3b0 at {username='slave'} at 127.27.196.69:41483: session id = c609c27f61b9436d95dc53deaa21a3b0-e5bca5fdd4d14b5baaa0b0cfc4754a23
I20240502 21:35:42.944058 26586 tablet_copy_source_session.cc:205] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:35:42.945791 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5bca5fdd4d14b5baaa0b0cfc4754a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:42.951054 28187 tablet_copy_client.cc:783] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P c609c27f61b9436d95dc53deaa21a3b0: tablet copy: Starting download of 0 data blocks...
I20240502 21:35:42.951753 28187 tablet_copy_client.cc:647] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P c609c27f61b9436d95dc53deaa21a3b0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:35:42.954005 28187 tablet_copy_client.cc:523] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P c609c27f61b9436d95dc53deaa21a3b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:35:42.956807 28187 tablet_bootstrap.cc:492] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P c609c27f61b9436d95dc53deaa21a3b0: Bootstrap starting.
I20240502 21:35:42.991645 28187 tablet_bootstrap.cc:492] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P c609c27f61b9436d95dc53deaa21a3b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:35:42.992071 28187 tablet_bootstrap.cc:492] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P c609c27f61b9436d95dc53deaa21a3b0: Bootstrap complete.
I20240502 21:35:42.992343 28187 ts_tablet_manager.cc:1367] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P c609c27f61b9436d95dc53deaa21a3b0: Time spent bootstrapping tablet: real 0.036s	user 0.033s	sys 0.002s
I20240502 21:35:42.993042 28187 raft_consensus.cc:348] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P c609c27f61b9436d95dc53deaa21a3b0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: true } }
I20240502 21:35:42.993283 28187 raft_consensus.cc:729] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P c609c27f61b9436d95dc53deaa21a3b0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c609c27f61b9436d95dc53deaa21a3b0, State: Initialized, Role: LEARNER
I20240502 21:35:42.993475 28187 consensus_queue.cc:260] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P c609c27f61b9436d95dc53deaa21a3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: NON_VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: true } }
I20240502 21:35:42.994299 28187 ts_tablet_manager.cc:1398] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P c609c27f61b9436d95dc53deaa21a3b0: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 21:35:42.995303 26586 tablet_copy_service.cc:342] P 5b1043ab39c04568814296d029600092: Request end of tablet copy session c609c27f61b9436d95dc53deaa21a3b0-e5bca5fdd4d14b5baaa0b0cfc4754a23 received from {username='slave'} at 127.27.196.69:41483
I20240502 21:35:42.995468 26586 tablet_copy_service.cc:433] P 5b1043ab39c04568814296d029600092: ending tablet copy session c609c27f61b9436d95dc53deaa21a3b0-e5bca5fdd4d14b5baaa0b0cfc4754a23 on tablet e5bca5fdd4d14b5baaa0b0cfc4754a23 with peer c609c27f61b9436d95dc53deaa21a3b0
I20240502 21:35:43.071339 28205 ts_tablet_manager.cc:902] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92: Initiating tablet copy from peer 63266faf827046629e39afa765b144bc (127.27.196.67:39203)
I20240502 21:35:43.072528 28205 tablet_copy_client.cc:308] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.67:39203
I20240502 21:35:43.073253 26720 tablet_copy_service.cc:140] P 63266faf827046629e39afa765b144bc: Received BeginTabletCopySession request for tablet 35508ded8421457288bdef00bf12f698 from peer 27be1297e75b4e20ba297e7332a98f92 ({username='slave'} at 127.27.196.68:47389)
I20240502 21:35:43.073441 26720 tablet_copy_service.cc:161] P 63266faf827046629e39afa765b144bc: Beginning new tablet copy session on tablet 35508ded8421457288bdef00bf12f698 from peer 27be1297e75b4e20ba297e7332a98f92 at {username='slave'} at 127.27.196.68:47389: session id = 27be1297e75b4e20ba297e7332a98f92-35508ded8421457288bdef00bf12f698
I20240502 21:35:43.075366 26720 tablet_copy_source_session.cc:205] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:35:43.076736 28205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35508ded8421457288bdef00bf12f698. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:43.081568 28205 tablet_copy_client.cc:783] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92: tablet copy: Starting download of 0 data blocks...
I20240502 21:35:43.081871 28205 tablet_copy_client.cc:647] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92: tablet copy: Starting download of 1 WAL segments...
I20240502 21:35:43.083701 28205 tablet_copy_client.cc:523] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:35:43.086270 28205 tablet_bootstrap.cc:492] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92: Bootstrap starting.
I20240502 21:35:43.119014 28205 tablet_bootstrap.cc:492] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:35:43.119560 28205 tablet_bootstrap.cc:492] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92: Bootstrap complete.
I20240502 21:35:43.119839 28205 ts_tablet_manager.cc:1367] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92: Time spent bootstrapping tablet: real 0.034s	user 0.027s	sys 0.008s
I20240502 21:35:43.120561 28205 raft_consensus.cc:348] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } }
I20240502 21:35:43.120790 28205 raft_consensus.cc:729] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 27be1297e75b4e20ba297e7332a98f92, State: Initialized, Role: LEARNER
I20240502 21:35:43.120999 28205 consensus_queue.cc:260] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: NON_VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: true } }
I20240502 21:35:43.121887 28205 ts_tablet_manager.cc:1398] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240502 21:35:43.122710 26720 tablet_copy_service.cc:342] P 63266faf827046629e39afa765b144bc: Request end of tablet copy session 27be1297e75b4e20ba297e7332a98f92-35508ded8421457288bdef00bf12f698 received from {username='slave'} at 127.27.196.68:47389
I20240502 21:35:43.122870 26720 tablet_copy_service.cc:433] P 63266faf827046629e39afa765b144bc: ending tablet copy session 27be1297e75b4e20ba297e7332a98f92-35508ded8421457288bdef00bf12f698 on tablet 35508ded8421457288bdef00bf12f698 with peer 27be1297e75b4e20ba297e7332a98f92
I20240502 21:35:43.366375 27375 raft_consensus.cc:1203] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P c609c27f61b9436d95dc53deaa21a3b0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:35:43.521103 27510 raft_consensus.cc:1203] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 21:35:43.867697 28248 raft_consensus.cc:1050] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092: attempting to promote NON_VOTER c609c27f61b9436d95dc53deaa21a3b0 to VOTER
I20240502 21:35:43.868436 28248 consensus_queue.cc:237] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } }
I20240502 21:35:43.870800 27375 raft_consensus.cc:1261] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P c609c27f61b9436d95dc53deaa21a3b0 [term 1 LEARNER]: Refusing update from remote peer 5b1043ab39c04568814296d029600092: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:35:43.871162 26698 raft_consensus.cc:1261] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Refusing update from remote peer 5b1043ab39c04568814296d029600092: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:35:43.871419 26430 raft_consensus.cc:1261] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Refusing update from remote peer 5b1043ab39c04568814296d029600092: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:35:43.871500 28224 consensus_queue.cc:1035] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:35:43.871898 28250 consensus_queue.cc:1035] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:35:43.872220 28248 consensus_queue.cc:1035] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:35:43.875320 28248 raft_consensus.cc:2932] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:43.876369 27375 raft_consensus.cc:2932] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P c609c27f61b9436d95dc53deaa21a3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:43.876435 26430 raft_consensus.cc:2932] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:43.880311 26698 raft_consensus.cc:2932] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:43.882068 26282 catalog_manager.cc:5484] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P c609c27f61b9436d95dc53deaa21a3b0 reported cstate change: config changed from index 4 to 5, c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b1043ab39c04568814296d029600092" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:43.919169 26565 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e5bca5fdd4d14b5baaa0b0cfc4754a23"
dest_uuid: "5b1043ab39c04568814296d029600092"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:50350
I20240502 21:35:43.919536 26565 raft_consensus.cc:595] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:35:43.992295 28237 raft_consensus.cc:1050] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc: attempting to promote NON_VOTER 27be1297e75b4e20ba297e7332a98f92 to VOTER
I20240502 21:35:43.993094 28237 consensus_queue.cc:237] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } }
I20240502 21:35:43.996387 26565 raft_consensus.cc:1261] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Refusing update from remote peer 63266faf827046629e39afa765b144bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:35:43.996385 26430 raft_consensus.cc:1261] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Refusing update from remote peer 63266faf827046629e39afa765b144bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:35:43.996467 27510 raft_consensus.cc:1261] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92 [term 1 LEARNER]: Refusing update from remote peer 63266faf827046629e39afa765b144bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:35:43.997095 28236 consensus_queue.cc:1035] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:35:43.997473 28278 consensus_queue.cc:1035] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:35:43.997829 28237 consensus_queue.cc:1035] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:35:44.005741 28236 raft_consensus.cc:2932] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:44.007885 26430 raft_consensus.cc:2932] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:44.009660 26565 raft_consensus.cc:2932] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:44.012310 26286 catalog_manager.cc:5484] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc reported cstate change: config changed from index 4 to 5, 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63266faf827046629e39afa765b144bc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:35:44.014947 27510 raft_consensus.cc:2932] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:44.144137 26698 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "35508ded8421457288bdef00bf12f698"
dest_uuid: "63266faf827046629e39afa765b144bc"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:39848
I20240502 21:35:44.144491 26698 raft_consensus.cc:595] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 1 LEADER]: Received request to transfer leadership
I20240502 21:35:44.168020 26565 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e5bca5fdd4d14b5baaa0b0cfc4754a23"
dest_uuid: "5b1043ab39c04568814296d029600092"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:50370
I20240502 21:35:44.168289 26565 raft_consensus.cc:595] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:35:44.285596 28251 raft_consensus.cc:979] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092: : Instructing follower 63266faf827046629e39afa765b144bc to start an election
I20240502 21:35:44.285807 28251 raft_consensus.cc:1067] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [term 1 LEADER]: Signalling peer 63266faf827046629e39afa765b144bc to start an election
I20240502 21:35:44.286949 26698 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e5bca5fdd4d14b5baaa0b0cfc4754a23"
dest_uuid: "63266faf827046629e39afa765b144bc"
 from {username='slave'} at 127.27.196.66:58287
I20240502 21:35:44.287300 26698 raft_consensus.cc:482] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:35:44.287448 26698 raft_consensus.cc:3037] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:35:44.289476 26698 raft_consensus.cc:504] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } }
I20240502 21:35:44.290351 26698 leader_election.cc:288] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc [CANDIDATE]: Term 2 election: Requested vote from peers 5b1043ab39c04568814296d029600092 (127.27.196.66:41285), cb27977c77564d38afe6e64c816eacd0 (127.27.196.65:36209), c609c27f61b9436d95dc53deaa21a3b0 (127.27.196.69:43269)
I20240502 21:35:44.290901 26565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5bca5fdd4d14b5baaa0b0cfc4754a23" candidate_uuid: "63266faf827046629e39afa765b144bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "5b1043ab39c04568814296d029600092"
I20240502 21:35:44.291078 26430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5bca5fdd4d14b5baaa0b0cfc4754a23" candidate_uuid: "63266faf827046629e39afa765b144bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "cb27977c77564d38afe6e64c816eacd0"
I20240502 21:35:44.291209 26565 raft_consensus.cc:3032] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:35:44.291357 26565 raft_consensus.cc:729] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5b1043ab39c04568814296d029600092, State: Running, Role: LEADER
I20240502 21:35:44.291431 26430 raft_consensus.cc:3037] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:35:44.291801 26565 consensus_queue.cc:260] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } }
I20240502 21:35:44.292263 26565 raft_consensus.cc:3037] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:35:44.294426 26430 raft_consensus.cc:2454] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P cb27977c77564d38afe6e64c816eacd0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63266faf827046629e39afa765b144bc in term 2.
I20240502 21:35:44.296074 26565 raft_consensus.cc:2454] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63266faf827046629e39afa765b144bc in term 2.
I20240502 21:35:44.296674 26632 leader_election.cc:302] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5b1043ab39c04568814296d029600092, 63266faf827046629e39afa765b144bc, cb27977c77564d38afe6e64c816eacd0; no voters: 
I20240502 21:35:44.297032 28236 raft_consensus.cc:2781] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:35:44.297211 28236 raft_consensus.cc:686] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc [term 2 LEADER]: Becoming Leader. State: Replica: 63266faf827046629e39afa765b144bc, State: Running, Role: LEADER
I20240502 21:35:44.297536 28236 consensus_queue.cc:237] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } }
I20240502 21:35:44.300469 26284 catalog_manager.cc:5484] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc reported cstate change: term changed from 1 to 2, leader changed from 5b1043ab39c04568814296d029600092 (127.27.196.66) to 63266faf827046629e39afa765b144bc (127.27.196.67). New cstate: current_term: 2 leader_uuid: "63266faf827046629e39afa765b144bc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:35:44.302455 27375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5bca5fdd4d14b5baaa0b0cfc4754a23" candidate_uuid: "63266faf827046629e39afa765b144bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c609c27f61b9436d95dc53deaa21a3b0"
I20240502 21:35:44.302809 27375 raft_consensus.cc:3037] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P c609c27f61b9436d95dc53deaa21a3b0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:35:44.305253 27375 raft_consensus.cc:2454] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P c609c27f61b9436d95dc53deaa21a3b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63266faf827046629e39afa765b144bc in term 2.
I20240502 21:35:44.395290 26697 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "35508ded8421457288bdef00bf12f698"
dest_uuid: "63266faf827046629e39afa765b144bc"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:39862
I20240502 21:35:44.395558 26697 raft_consensus.cc:595] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 1 LEADER]: Received request to transfer leadership
I20240502 21:35:44.403669 28278 raft_consensus.cc:979] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc: : Instructing follower cb27977c77564d38afe6e64c816eacd0 to start an election
I20240502 21:35:44.403950 28236 raft_consensus.cc:1067] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 1 LEADER]: Signalling peer cb27977c77564d38afe6e64c816eacd0 to start an election
I20240502 21:35:44.405128 26430 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "35508ded8421457288bdef00bf12f698"
dest_uuid: "cb27977c77564d38afe6e64c816eacd0"
 from {username='slave'} at 127.27.196.67:40999
I20240502 21:35:44.405426 26430 raft_consensus.cc:482] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:35:44.405535 26430 raft_consensus.cc:3037] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:35:44.408078 26430 raft_consensus.cc:504] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } }
I20240502 21:35:44.409499 26565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35508ded8421457288bdef00bf12f698" candidate_uuid: "cb27977c77564d38afe6e64c816eacd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "5b1043ab39c04568814296d029600092"
I20240502 21:35:44.409761 26565 raft_consensus.cc:3037] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:35:44.410311 26430 leader_election.cc:288] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [CANDIDATE]: Term 2 election: Requested vote from peers 5b1043ab39c04568814296d029600092 (127.27.196.66:41285), 63266faf827046629e39afa765b144bc (127.27.196.67:39203), 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68:35953)
I20240502 21:35:44.410705 26697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35508ded8421457288bdef00bf12f698" candidate_uuid: "cb27977c77564d38afe6e64c816eacd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "63266faf827046629e39afa765b144bc"
I20240502 21:35:44.410913 26697 raft_consensus.cc:3032] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:35:44.410991 26697 raft_consensus.cc:729] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 63266faf827046629e39afa765b144bc, State: Running, Role: LEADER
I20240502 21:35:44.411538 26697 consensus_queue.cc:260] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } }
I20240502 21:35:44.411923 26697 raft_consensus.cc:3037] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:35:44.412281 26565 raft_consensus.cc:2454] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb27977c77564d38afe6e64c816eacd0 in term 2.
I20240502 21:35:44.412894 27510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35508ded8421457288bdef00bf12f698" candidate_uuid: "cb27977c77564d38afe6e64c816eacd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "27be1297e75b4e20ba297e7332a98f92"
I20240502 21:35:44.413218 27510 raft_consensus.cc:3037] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:35:44.413962 26697 raft_consensus.cc:2454] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb27977c77564d38afe6e64c816eacd0 in term 2.
I20240502 21:35:44.414450 26364 leader_election.cc:302] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5b1043ab39c04568814296d029600092, 63266faf827046629e39afa765b144bc, cb27977c77564d38afe6e64c816eacd0; no voters: 
I20240502 21:35:44.414808 27925 raft_consensus.cc:2781] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:35:44.415292 27925 raft_consensus.cc:686] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 2 LEADER]: Becoming Leader. State: Replica: cb27977c77564d38afe6e64c816eacd0, State: Running, Role: LEADER
I20240502 21:35:44.415705 27510 raft_consensus.cc:2454] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb27977c77564d38afe6e64c816eacd0 in term 2.
I20240502 21:35:44.415719 27925 consensus_queue.cc:237] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } }
I20240502 21:35:44.419042 26284 catalog_manager.cc:5484] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 reported cstate change: term changed from 1 to 2, leader changed from 63266faf827046629e39afa765b144bc (127.27.196.67) to cb27977c77564d38afe6e64c816eacd0 (127.27.196.65). New cstate: current_term: 2 leader_uuid: "cb27977c77564d38afe6e64c816eacd0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:35:44.684370 26430 raft_consensus.cc:1261] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P cb27977c77564d38afe6e64c816eacd0 [term 2 FOLLOWER]: Refusing update from remote peer 63266faf827046629e39afa765b144bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:35:44.685086 28236 consensus_queue.cc:1035] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:35:44.823431 27375 raft_consensus.cc:1261] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P c609c27f61b9436d95dc53deaa21a3b0 [term 2 FOLLOWER]: Refusing update from remote peer 63266faf827046629e39afa765b144bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:35:44.824383 28236 consensus_queue.cc:1035] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:35:44.827750 26565 raft_consensus.cc:1261] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092 [term 2 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:35:44.829386 27925 consensus_queue.cc:1035] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:35:44.830353 26565 raft_consensus.cc:1261] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [term 2 FOLLOWER]: Refusing update from remote peer 63266faf827046629e39afa765b144bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:35:44.833161 28236 consensus_queue.cc:1035] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:35:44.841475 26697 consensus_queue.cc:237] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } }
I20240502 21:35:44.844090 27375 raft_consensus.cc:1261] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P c609c27f61b9436d95dc53deaa21a3b0 [term 2 FOLLOWER]: Refusing update from remote peer 63266faf827046629e39afa765b144bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:35:44.844614 26430 raft_consensus.cc:1261] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P cb27977c77564d38afe6e64c816eacd0 [term 2 FOLLOWER]: Refusing update from remote peer 63266faf827046629e39afa765b144bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:35:44.845433 28336 consensus_queue.cc:1035] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:35:44.846124 28336 consensus_queue.cc:1035] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:35:44.849126 28336 raft_consensus.cc:2932] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5b1043ab39c04568814296d029600092 (127.27.196.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:44.849856 27375 raft_consensus.cc:2932] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P c609c27f61b9436d95dc53deaa21a3b0 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5b1043ab39c04568814296d029600092 (127.27.196.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:44.851859 26430 raft_consensus.cc:2932] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P cb27977c77564d38afe6e64c816eacd0 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5b1043ab39c04568814296d029600092 (127.27.196.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } } }
I20240502 21:35:44.854462 26274 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e5bca5fdd4d14b5baaa0b0cfc4754a23 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:35:44.859932 26287 catalog_manager.cc:5484] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc reported cstate change: config changed from index 5 to 7, VOTER 5b1043ab39c04568814296d029600092 (127.27.196.66) evicted. New cstate: current_term: 2 leader_uuid: "63266faf827046629e39afa765b144bc" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c609c27f61b9436d95dc53deaa21a3b0" member_type: VOTER last_known_addr { host: "127.27.196.69" port: 43269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:35:44.865074 26543 tablet_service.cc:1514] Processing DeleteTablet for tablet e5bca5fdd4d14b5baaa0b0cfc4754a23 with delete_type TABLET_DATA_TOMBSTONED (TS 5b1043ab39c04568814296d029600092 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34904
I20240502 21:35:44.867679 27510 raft_consensus.cc:1261] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92 [term 2 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:35:44.867935 28348 tablet_replica.cc:330] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092: stopping tablet replica
I20240502 21:35:44.868163 28348 raft_consensus.cc:2229] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:35:44.868392 28348 raft_consensus.cc:2258] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:35:44.868479 27925 consensus_queue.cc:1035] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:35:44.869750 28348 ts_tablet_manager.cc:1875] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:35:44.877686 28348 ts_tablet_manager.cc:1888] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 21:35:44.877718 26697 raft_consensus.cc:1261] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 2 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:35:44.877869 28348 log.cc:1196] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 5b1043ab39c04568814296d029600092: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/wals/e5bca5fdd4d14b5baaa0b0cfc4754a23
I20240502 21:35:44.878664 27925 consensus_queue.cc:1035] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:35:44.889379 26273 catalog_manager.cc:4830] TS 5b1043ab39c04568814296d029600092 (127.27.196.66:41285): tablet e5bca5fdd4d14b5baaa0b0cfc4754a23 (table rebalance_test_table_2 [id=79000138ee6c4b96965a29a347143ffa]) successfully deleted
I20240502 21:35:44.896349 26430 consensus_queue.cc:237] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } }
I20240502 21:35:44.899066 26564 raft_consensus.cc:1261] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092 [term 2 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:35:44.899235 27509 raft_consensus.cc:1261] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92 [term 2 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:35:44.899823 28350 consensus_queue.cc:1035] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:35:44.900440 28319 consensus_queue.cc:1035] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:35:44.902781 27925 raft_consensus.cc:2932] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 63266faf827046629e39afa765b144bc (127.27.196.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:44.903471 26564 raft_consensus.cc:2932] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 63266faf827046629e39afa765b144bc (127.27.196.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:44.907327 26273 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 35508ded8421457288bdef00bf12f698 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:35:44.908298 27509 raft_consensus.cc:2932] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 63266faf827046629e39afa765b144bc (127.27.196.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } }
I20240502 21:35:44.909202 26286 catalog_manager.cc:5484] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 reported cstate change: config changed from index 5 to 7, VOTER 63266faf827046629e39afa765b144bc (127.27.196.67) evicted. New cstate: current_term: 2 leader_uuid: "cb27977c77564d38afe6e64c816eacd0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:35:44.914237 26678 tablet_service.cc:1514] Processing DeleteTablet for tablet 35508ded8421457288bdef00bf12f698 with delete_type TABLET_DATA_TOMBSTONED (TS 63266faf827046629e39afa765b144bc not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:42438
I20240502 21:35:44.915133 28359 tablet_replica.cc:330] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc: stopping tablet replica
I20240502 21:35:44.915620 28359 raft_consensus.cc:2229] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:35:44.915956 28359 raft_consensus.cc:2258] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:35:44.917651 28359 ts_tablet_manager.cc:1875] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:35:44.923993 28359 ts_tablet_manager.cc:1888] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 21:35:44.924166 28359 log.cc:1196] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/wals/35508ded8421457288bdef00bf12f698
I20240502 21:35:44.925108 26274 catalog_manager.cc:4830] TS 63266faf827046629e39afa765b144bc (127.27.196.67:39203): tablet 35508ded8421457288bdef00bf12f698 (table rebalance_test_table_1 [id=cc5055b898c74d0ea2d82c8d8f1f5bb4]) successfully deleted
I20240502 21:35:45.045842 26678 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:45.067858 26543 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:45.079056 27490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:35:45.092123 27354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:35:45.106374 26409 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:35:45.306921 27490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:35:45.309082 26678 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:45.313024 26543 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:45.324839 26409 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:35:45.328450 27354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:35:45.412977 26430 consensus_queue.cc:237] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: NON_VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: true } }
I20240502 21:35:45.415570 26564 raft_consensus.cc:1261] T 140f7280768e4ad592309e9b7d6a26da P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:35:45.415606 27509 raft_consensus.cc:1261] T 140f7280768e4ad592309e9b7d6a26da P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 21:35:45.415818 26364 consensus_peers.cc:485] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 -> Peer 63266faf827046629e39afa765b144bc (127.27.196.67:39203): Couldn't send request to peer 63266faf827046629e39afa765b144bc. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:45.416276 28350 consensus_queue.cc:1035] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:35:45.416769 28443 consensus_queue.cc:1035] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:35:45.420974 28350 raft_consensus.cc:2932] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 63266faf827046629e39afa765b144bc (127.27.196.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: NON_VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: true } } }
I20240502 21:35:45.422061 26564 raft_consensus.cc:2932] T 140f7280768e4ad592309e9b7d6a26da P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 63266faf827046629e39afa765b144bc (127.27.196.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: NON_VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: true } } }
I20240502 21:35:45.422194 27509 raft_consensus.cc:2932] T 140f7280768e4ad592309e9b7d6a26da P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 63266faf827046629e39afa765b144bc (127.27.196.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: NON_VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: true } } }
I20240502 21:35:45.426474 26286 catalog_manager.cc:5484] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 reported cstate change: config changed from index 6 to 7, NON_VOTER 63266faf827046629e39afa765b144bc (127.27.196.67) added. New cstate: current_term: 1 leader_uuid: "cb27977c77564d38afe6e64c816eacd0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: NON_VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:35:45.448216 26430 consensus_queue.cc:237] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: NON_VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: true } }
I20240502 21:35:45.450734 27509 raft_consensus.cc:1261] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92 [term 2 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 21:35:45.450712 26564 raft_consensus.cc:1261] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092 [term 2 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240502 21:35:45.450868 26364 consensus_peers.cc:485] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 -> Peer 63266faf827046629e39afa765b144bc (127.27.196.67:39203): Couldn't send request to peer 63266faf827046629e39afa765b144bc. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:45.451359 27925 consensus_queue.cc:1035] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:35:45.451705 28350 consensus_queue.cc:1035] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:35:45.454950 28443 raft_consensus.cc:2932] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 63266faf827046629e39afa765b144bc (127.27.196.67) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: NON_VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: true } } }
I20240502 21:35:45.455783 26564 raft_consensus.cc:2932] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 63266faf827046629e39afa765b144bc (127.27.196.67) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: NON_VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: true } } }
I20240502 21:35:45.456120 27509 raft_consensus.cc:2932] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 63266faf827046629e39afa765b144bc (127.27.196.67) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: NON_VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: true } } }
I20240502 21:35:45.461839 26287 catalog_manager.cc:5484] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092 reported cstate change: config changed from index 7 to 8, NON_VOTER 63266faf827046629e39afa765b144bc (127.27.196.67) added. New cstate: current_term: 2 leader_uuid: "cb27977c77564d38afe6e64c816eacd0" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: NON_VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: true } } }
I20240502 21:35:45.483980 26430 consensus_queue.cc:237] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: NON_VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { promote: true } }
I20240502 21:35:45.486510 26697 raft_consensus.cc:1261] T d5cf433249bc4c309c56c9d2e02efebe P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:35:45.486532 27509 raft_consensus.cc:1261] T d5cf433249bc4c309c56c9d2e02efebe P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 21:35:45.486944 26363 consensus_peers.cc:485] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 -> Peer 5b1043ab39c04568814296d029600092 (127.27.196.66:41285): Couldn't send request to peer 5b1043ab39c04568814296d029600092. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:45.487311 28350 consensus_queue.cc:1035] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:35:45.487682 28443 consensus_queue.cc:1035] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:35:45.490316 28350 raft_consensus.cc:2932] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5b1043ab39c04568814296d029600092 (127.27.196.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: NON_VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { promote: true } } }
I20240502 21:35:45.491261 27509 raft_consensus.cc:2932] T d5cf433249bc4c309c56c9d2e02efebe P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5b1043ab39c04568814296d029600092 (127.27.196.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: NON_VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { promote: true } } }
I20240502 21:35:45.491856 26697 raft_consensus.cc:2932] T d5cf433249bc4c309c56c9d2e02efebe P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5b1043ab39c04568814296d029600092 (127.27.196.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: NON_VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { promote: true } } }
I20240502 21:35:45.500272 26286 catalog_manager.cc:5484] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 reported cstate change: config changed from index 6 to 7, NON_VOTER 5b1043ab39c04568814296d029600092 (127.27.196.66) added. New cstate: current_term: 1 leader_uuid: "cb27977c77564d38afe6e64c816eacd0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: NON_VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:35:45.523658 26564 consensus_queue.cc:237] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: NON_VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { promote: true } }
I20240502 21:35:45.527371 27509 raft_consensus.cc:1261] T 8747e6794c084510a5b9e1f34c15c733 P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Refusing update from remote peer 5b1043ab39c04568814296d029600092: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 21:35:45.527534 26498 consensus_peers.cc:485] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 -> Peer cb27977c77564d38afe6e64c816eacd0 (127.27.196.65:36209): Couldn't send request to peer cb27977c77564d38afe6e64c816eacd0. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:45.528159 26697 raft_consensus.cc:1261] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Refusing update from remote peer 5b1043ab39c04568814296d029600092: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:35:45.528180 28251 consensus_queue.cc:1035] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:35:45.528779 28469 consensus_queue.cc:1035] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:35:45.532078 28251 raft_consensus.cc:2932] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER cb27977c77564d38afe6e64c816eacd0 (127.27.196.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: NON_VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { promote: true } } }
I20240502 21:35:45.533569 27509 raft_consensus.cc:2932] T 8747e6794c084510a5b9e1f34c15c733 P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER cb27977c77564d38afe6e64c816eacd0 (127.27.196.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: NON_VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { promote: true } } }
I20240502 21:35:45.534255 26697 raft_consensus.cc:2932] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER cb27977c77564d38afe6e64c816eacd0 (127.27.196.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: NON_VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { promote: true } } }
I20240502 21:35:45.540599 26282 catalog_manager.cc:5484] T 8747e6794c084510a5b9e1f34c15c733 P 27be1297e75b4e20ba297e7332a98f92 reported cstate change: config changed from index 6 to 7, NON_VOTER cb27977c77564d38afe6e64c816eacd0 (127.27.196.65) added. New cstate: current_term: 1 leader_uuid: "5b1043ab39c04568814296d029600092" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: NON_VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { promote: true } } }
I20240502 21:35:45.877141 28517 ts_tablet_manager.cc:902] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc: Initiating tablet copy from peer cb27977c77564d38afe6e64c816eacd0 (127.27.196.65:36209)
I20240502 21:35:45.878729 28517 tablet_copy_client.cc:308] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:36209
I20240502 21:35:45.879735 26450 tablet_copy_service.cc:140] P cb27977c77564d38afe6e64c816eacd0: Received BeginTabletCopySession request for tablet 35508ded8421457288bdef00bf12f698 from peer 63266faf827046629e39afa765b144bc ({username='slave'} at 127.27.196.67:40999)
I20240502 21:35:45.879894 26450 tablet_copy_service.cc:161] P cb27977c77564d38afe6e64c816eacd0: Beginning new tablet copy session on tablet 35508ded8421457288bdef00bf12f698 from peer 63266faf827046629e39afa765b144bc at {username='slave'} at 127.27.196.67:40999: session id = 63266faf827046629e39afa765b144bc-35508ded8421457288bdef00bf12f698
I20240502 21:35:45.881974 26450 tablet_copy_source_session.cc:205] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:35:45.883488 28517 ts_tablet_manager.cc:1875] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:35:45.889259 28517 ts_tablet_manager.cc:1888] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240502 21:35:45.889560 28517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35508ded8421457288bdef00bf12f698. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:45.892166 28517 tablet_copy_client.cc:783] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc: tablet copy: Starting download of 0 data blocks...
I20240502 21:35:45.892463 28517 tablet_copy_client.cc:647] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc: tablet copy: Starting download of 1 WAL segments...
I20240502 21:35:45.894604 28517 tablet_copy_client.cc:523] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:35:45.897676 28517 tablet_bootstrap.cc:492] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc: Bootstrap starting.
I20240502 21:35:45.932469 28517 tablet_bootstrap.cc:492] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:35:45.932950 28517 tablet_bootstrap.cc:492] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc: Bootstrap complete.
I20240502 21:35:45.933228 28517 ts_tablet_manager.cc:1367] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc: Time spent bootstrapping tablet: real 0.036s	user 0.032s	sys 0.004s
I20240502 21:35:45.933923 28517 raft_consensus.cc:348] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: NON_VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: true } }
I20240502 21:35:45.934154 28517 raft_consensus.cc:729] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 63266faf827046629e39afa765b144bc, State: Initialized, Role: LEARNER
I20240502 21:35:45.934342 28517 consensus_queue.cc:260] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: NON_VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: true } }
I20240502 21:35:45.934898 28517 ts_tablet_manager.cc:1398] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 21:35:45.935916 26450 tablet_copy_service.cc:342] P cb27977c77564d38afe6e64c816eacd0: Request end of tablet copy session 63266faf827046629e39afa765b144bc-35508ded8421457288bdef00bf12f698 received from {username='slave'} at 127.27.196.67:40999
I20240502 21:35:45.936074 26450 tablet_copy_service.cc:433] P cb27977c77564d38afe6e64c816eacd0: ending tablet copy session 63266faf827046629e39afa765b144bc-35508ded8421457288bdef00bf12f698 on tablet 35508ded8421457288bdef00bf12f698 with peer 63266faf827046629e39afa765b144bc
I20240502 21:35:45.959038 28523 ts_tablet_manager.cc:902] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092: Initiating tablet copy from peer cb27977c77564d38afe6e64c816eacd0 (127.27.196.65:36209)
I20240502 21:35:45.961001 28523 tablet_copy_client.cc:308] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:36209
I20240502 21:35:45.961835 26450 tablet_copy_service.cc:140] P cb27977c77564d38afe6e64c816eacd0: Received BeginTabletCopySession request for tablet d5cf433249bc4c309c56c9d2e02efebe from peer 5b1043ab39c04568814296d029600092 ({username='slave'} at 127.27.196.66:45063)
I20240502 21:35:45.962030 26450 tablet_copy_service.cc:161] P cb27977c77564d38afe6e64c816eacd0: Beginning new tablet copy session on tablet d5cf433249bc4c309c56c9d2e02efebe from peer 5b1043ab39c04568814296d029600092 at {username='slave'} at 127.27.196.66:45063: session id = 5b1043ab39c04568814296d029600092-d5cf433249bc4c309c56c9d2e02efebe
I20240502 21:35:45.964246 26450 tablet_copy_source_session.cc:205] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:35:45.965569 28523 ts_tablet_manager.cc:1875] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:35:45.970885 28523 ts_tablet_manager.cc:1888] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 21:35:45.971204 28523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5cf433249bc4c309c56c9d2e02efebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:45.973652 28523 tablet_copy_client.cc:783] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092: tablet copy: Starting download of 0 data blocks...
I20240502 21:35:45.973942 28523 tablet_copy_client.cc:647] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092: tablet copy: Starting download of 1 WAL segments...
I20240502 21:35:45.975955 28523 tablet_copy_client.cc:523] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:35:45.979182 28523 tablet_bootstrap.cc:492] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092: Bootstrap starting.
I20240502 21:35:46.016939 28523 tablet_bootstrap.cc:492] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:35:46.017464 28523 tablet_bootstrap.cc:492] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092: Bootstrap complete.
I20240502 21:35:46.017762 28523 ts_tablet_manager.cc:1367] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092: Time spent bootstrapping tablet: real 0.039s	user 0.038s	sys 0.000s
I20240502 21:35:46.018503 28523 raft_consensus.cc:348] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: NON_VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { promote: true } }
I20240502 21:35:46.018745 28523 raft_consensus.cc:729] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b1043ab39c04568814296d029600092, State: Initialized, Role: LEARNER
I20240502 21:35:46.018937 28523 consensus_queue.cc:260] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: NON_VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { promote: true } }
I20240502 21:35:46.019536 28523 ts_tablet_manager.cc:1398] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240502 21:35:46.020893 26450 tablet_copy_service.cc:342] P cb27977c77564d38afe6e64c816eacd0: Request end of tablet copy session 5b1043ab39c04568814296d029600092-d5cf433249bc4c309c56c9d2e02efebe received from {username='slave'} at 127.27.196.66:45063
I20240502 21:35:46.021021 26450 tablet_copy_service.cc:433] P cb27977c77564d38afe6e64c816eacd0: ending tablet copy session 5b1043ab39c04568814296d029600092-d5cf433249bc4c309c56c9d2e02efebe on tablet d5cf433249bc4c309c56c9d2e02efebe with peer 5b1043ab39c04568814296d029600092
I20240502 21:35:46.028522 28529 ts_tablet_manager.cc:902] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0: Initiating tablet copy from peer 5b1043ab39c04568814296d029600092 (127.27.196.66:41285)
I20240502 21:35:46.030115 28529 tablet_copy_client.cc:308] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.66:41285
I20240502 21:35:46.031060 26586 tablet_copy_service.cc:140] P 5b1043ab39c04568814296d029600092: Received BeginTabletCopySession request for tablet 8747e6794c084510a5b9e1f34c15c733 from peer cb27977c77564d38afe6e64c816eacd0 ({username='slave'} at 127.27.196.65:36713)
I20240502 21:35:46.031302 26586 tablet_copy_service.cc:161] P 5b1043ab39c04568814296d029600092: Beginning new tablet copy session on tablet 8747e6794c084510a5b9e1f34c15c733 from peer cb27977c77564d38afe6e64c816eacd0 at {username='slave'} at 127.27.196.65:36713: session id = cb27977c77564d38afe6e64c816eacd0-8747e6794c084510a5b9e1f34c15c733
I20240502 21:35:46.033411 26586 tablet_copy_source_session.cc:205] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:35:46.034179 28517 ts_tablet_manager.cc:902] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc: Initiating tablet copy from peer cb27977c77564d38afe6e64c816eacd0 (127.27.196.65:36209)
I20240502 21:35:46.035053 28529 ts_tablet_manager.cc:1875] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:35:46.035534 28517 tablet_copy_client.cc:308] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc: tablet copy: Beginning tablet copy session from remote peer at address 127.27.196.65:36209
I20240502 21:35:46.036253 26450 tablet_copy_service.cc:140] P cb27977c77564d38afe6e64c816eacd0: Received BeginTabletCopySession request for tablet 140f7280768e4ad592309e9b7d6a26da from peer 63266faf827046629e39afa765b144bc ({username='slave'} at 127.27.196.67:40999)
I20240502 21:35:46.036412 26450 tablet_copy_service.cc:161] P cb27977c77564d38afe6e64c816eacd0: Beginning new tablet copy session on tablet 140f7280768e4ad592309e9b7d6a26da from peer 63266faf827046629e39afa765b144bc at {username='slave'} at 127.27.196.67:40999: session id = 63266faf827046629e39afa765b144bc-140f7280768e4ad592309e9b7d6a26da
I20240502 21:35:46.038727 26450 tablet_copy_source_session.cc:205] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:35:46.039800 28517 ts_tablet_manager.cc:1875] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:35:46.040650 28529 ts_tablet_manager.cc:1888] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 21:35:46.040937 28529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8747e6794c084510a5b9e1f34c15c733. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:46.043663 28529 tablet_copy_client.cc:783] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0: tablet copy: Starting download of 0 data blocks...
I20240502 21:35:46.044060 28529 tablet_copy_client.cc:647] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:35:46.044638 28517 ts_tablet_manager.cc:1888] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 21:35:46.044934 28517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 140f7280768e4ad592309e9b7d6a26da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:46.046361 28529 tablet_copy_client.cc:523] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:35:46.049024 28517 tablet_copy_client.cc:783] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc: tablet copy: Starting download of 0 data blocks...
I20240502 21:35:46.049294 28517 tablet_copy_client.cc:647] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc: tablet copy: Starting download of 1 WAL segments...
I20240502 21:35:46.049919 28529 tablet_bootstrap.cc:492] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0: Bootstrap starting.
I20240502 21:35:46.051169 28517 tablet_copy_client.cc:523] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:35:46.053598 28517 tablet_bootstrap.cc:492] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc: Bootstrap starting.
I20240502 21:35:46.109143 28529 tablet_bootstrap.cc:492] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:35:46.109714 28529 tablet_bootstrap.cc:492] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0: Bootstrap complete.
I20240502 21:35:46.110023 28529 ts_tablet_manager.cc:1367] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0: Time spent bootstrapping tablet: real 0.060s	user 0.053s	sys 0.004s
I20240502 21:35:46.110913 28529 raft_consensus.cc:348] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: NON_VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { promote: true } }
I20240502 21:35:46.111243 28529 raft_consensus.cc:729] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cb27977c77564d38afe6e64c816eacd0, State: Initialized, Role: LEARNER
I20240502 21:35:46.111649 28529 consensus_queue.cc:260] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: NON_VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { promote: true } }
I20240502 21:35:46.113469 28529 ts_tablet_manager.cc:1398] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0: Time spent starting tablet: real 0.003s	user 0.005s	sys 0.000s
I20240502 21:35:46.115849 26586 tablet_copy_service.cc:342] P 5b1043ab39c04568814296d029600092: Request end of tablet copy session cb27977c77564d38afe6e64c816eacd0-8747e6794c084510a5b9e1f34c15c733 received from {username='slave'} at 127.27.196.65:36713
I20240502 21:35:46.116003 26586 tablet_copy_service.cc:433] P 5b1043ab39c04568814296d029600092: ending tablet copy session cb27977c77564d38afe6e64c816eacd0-8747e6794c084510a5b9e1f34c15c733 on tablet 8747e6794c084510a5b9e1f34c15c733 with peer cb27977c77564d38afe6e64c816eacd0
I20240502 21:35:46.120582 28517 tablet_bootstrap.cc:492] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:35:46.121079 28517 tablet_bootstrap.cc:492] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc: Bootstrap complete.
I20240502 21:35:46.121351 28517 ts_tablet_manager.cc:1367] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc: Time spent bootstrapping tablet: real 0.068s	user 0.058s	sys 0.008s
I20240502 21:35:46.122283 28517 raft_consensus.cc:348] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: NON_VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: true } }
I20240502 21:35:46.122502 28517 raft_consensus.cc:729] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63266faf827046629e39afa765b144bc, State: Initialized, Role: LEARNER
I20240502 21:35:46.122741 28517 consensus_queue.cc:260] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: NON_VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: true } }
I20240502 21:35:46.123502 28517 ts_tablet_manager.cc:1398] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 21:35:46.124917 26450 tablet_copy_service.cc:342] P cb27977c77564d38afe6e64c816eacd0: Request end of tablet copy session 63266faf827046629e39afa765b144bc-140f7280768e4ad592309e9b7d6a26da received from {username='slave'} at 127.27.196.67:40999
I20240502 21:35:46.125084 26450 tablet_copy_service.cc:433] P cb27977c77564d38afe6e64c816eacd0: ending tablet copy session 63266faf827046629e39afa765b144bc-140f7280768e4ad592309e9b7d6a26da on tablet 140f7280768e4ad592309e9b7d6a26da with peer 63266faf827046629e39afa765b144bc
I20240502 21:35:46.349382 26697 raft_consensus.cc:1203] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8]   Dedup: 2.8->[]
I20240502 21:35:46.473407 26564 raft_consensus.cc:1203] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 21:35:46.494161 26430 raft_consensus.cc:1203] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 21:35:46.598520 26697 raft_consensus.cc:1203] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 21:35:46.925918 27925 raft_consensus.cc:1050] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0: attempting to promote NON_VOTER 63266faf827046629e39afa765b144bc to VOTER
I20240502 21:35:46.926709 27925 consensus_queue.cc:237] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false } }
I20240502 21:35:46.929986 26697 raft_consensus.cc:1261] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 2 LEARNER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:35:46.930090 27509 raft_consensus.cc:1261] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92 [term 2 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:35:46.930459 26564 raft_consensus.cc:1261] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092 [term 2 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:35:46.930800 28628 consensus_queue.cc:1035] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:35:46.931430 27925 consensus_queue.cc:1035] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:35:46.931952 28561 consensus_queue.cc:1035] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:35:46.936092 28561 raft_consensus.cc:2932] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 63266faf827046629e39afa765b144bc (127.27.196.67) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false } } }
I20240502 21:35:46.937032 26697 raft_consensus.cc:2932] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 63266faf827046629e39afa765b144bc (127.27.196.67) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false } } }
I20240502 21:35:46.939571 27509 raft_consensus.cc:2932] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 63266faf827046629e39afa765b144bc (127.27.196.67) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false } } }
I20240502 21:35:46.942410 26564 raft_consensus.cc:2932] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 63266faf827046629e39afa765b144bc (127.27.196.67) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false } } }
I20240502 21:35:46.943388 26284 catalog_manager.cc:5484] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 reported cstate change: config changed from index 8 to 9, 63266faf827046629e39afa765b144bc (127.27.196.67) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "cb27977c77564d38afe6e64c816eacd0" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:35:47.088215 28637 raft_consensus.cc:1050] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092: attempting to promote NON_VOTER cb27977c77564d38afe6e64c816eacd0 to VOTER
I20240502 21:35:47.088994 28637 consensus_queue.cc:237] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { promote: false } }
I20240502 21:35:47.091707 26430 raft_consensus.cc:1261] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0 [term 1 LEARNER]: Refusing update from remote peer 5b1043ab39c04568814296d029600092: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:35:47.091763 27509 raft_consensus.cc:1261] T 8747e6794c084510a5b9e1f34c15c733 P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Refusing update from remote peer 5b1043ab39c04568814296d029600092: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:35:47.091763 26697 raft_consensus.cc:1261] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Refusing update from remote peer 5b1043ab39c04568814296d029600092: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:35:47.092377 28640 consensus_queue.cc:1035] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:35:47.092851 28637 consensus_queue.cc:1035] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:35:47.093214 28583 consensus_queue.cc:1035] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:35:47.096694 28583 raft_consensus.cc:2932] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cb27977c77564d38afe6e64c816eacd0 (127.27.196.65) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { promote: false } } }
I20240502 21:35:47.097640 27509 raft_consensus.cc:2932] T 8747e6794c084510a5b9e1f34c15c733 P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cb27977c77564d38afe6e64c816eacd0 (127.27.196.65) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { promote: false } } }
I20240502 21:35:47.097599 26697 raft_consensus.cc:2932] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cb27977c77564d38afe6e64c816eacd0 (127.27.196.65) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { promote: false } } }
I20240502 21:35:47.102658 26430 raft_consensus.cc:2932] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cb27977c77564d38afe6e64c816eacd0 (127.27.196.65) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { promote: false } } }
I20240502 21:35:47.103900 28628 raft_consensus.cc:1050] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0: attempting to promote NON_VOTER 5b1043ab39c04568814296d029600092 to VOTER
I20240502 21:35:47.104964 28628 consensus_queue.cc:237] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { promote: false } }
I20240502 21:35:47.107463 26287 catalog_manager.cc:5484] T 8747e6794c084510a5b9e1f34c15c733 P 27be1297e75b4e20ba297e7332a98f92 reported cstate change: config changed from index 7 to 8, cb27977c77564d38afe6e64c816eacd0 (127.27.196.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b1043ab39c04568814296d029600092" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { promote: false } } }
I20240502 21:35:47.111128 26564 raft_consensus.cc:1261] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092 [term 1 LEARNER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:35:47.111402 26697 raft_consensus.cc:1261] T d5cf433249bc4c309c56c9d2e02efebe P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:35:47.111404 27509 raft_consensus.cc:1261] T d5cf433249bc4c309c56c9d2e02efebe P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:35:47.112212 27925 consensus_queue.cc:1035] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:35:47.112654 28628 consensus_queue.cc:1035] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:35:47.113034 28560 consensus_queue.cc:1035] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:35:47.122658 26565 consensus_queue.cc:237] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { promote: false } }
I20240502 21:35:47.127069 27510 raft_consensus.cc:1261] T 8747e6794c084510a5b9e1f34c15c733 P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Refusing update from remote peer 5b1043ab39c04568814296d029600092: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:35:47.127821 26430 raft_consensus.cc:1261] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Refusing update from remote peer 5b1043ab39c04568814296d029600092: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:35:47.129163 28637 consensus_queue.cc:1035] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:35:47.130101 28637 consensus_queue.cc:1035] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:35:47.133816 28583 raft_consensus.cc:2932] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 63266faf827046629e39afa765b144bc (127.27.196.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { promote: false } } }
I20240502 21:35:47.136153 27510 raft_consensus.cc:2932] T 8747e6794c084510a5b9e1f34c15c733 P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 63266faf827046629e39afa765b144bc (127.27.196.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { promote: false } } }
I20240502 21:35:47.139387 28560 raft_consensus.cc:2932] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5b1043ab39c04568814296d029600092 (127.27.196.66) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { promote: false } } }
I20240502 21:35:47.142701 26286 catalog_manager.cc:5484] T 8747e6794c084510a5b9e1f34c15c733 P 5b1043ab39c04568814296d029600092 reported cstate change: config changed from index 8 to 9, VOTER 63266faf827046629e39afa765b144bc (127.27.196.67) evicted. New cstate: current_term: 1 leader_uuid: "5b1043ab39c04568814296d029600092" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:35:47.145704 27510 raft_consensus.cc:2932] T d5cf433249bc4c309c56c9d2e02efebe P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5b1043ab39c04568814296d029600092 (127.27.196.66) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { promote: false } } }
I20240502 21:35:47.147137 26430 raft_consensus.cc:2932] T 8747e6794c084510a5b9e1f34c15c733 P cb27977c77564d38afe6e64c816eacd0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 63266faf827046629e39afa765b144bc (127.27.196.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { promote: false } } }
I20240502 21:35:47.150156 26678 tablet_service.cc:1514] Processing DeleteTablet for tablet 8747e6794c084510a5b9e1f34c15c733 with delete_type TABLET_DATA_TOMBSTONED (TS 63266faf827046629e39afa765b144bc not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:42438
I20240502 21:35:47.150701 26273 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8747e6794c084510a5b9e1f34c15c733 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:35:47.154683 26564 raft_consensus.cc:2932] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5b1043ab39c04568814296d029600092 (127.27.196.66) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { promote: false } } }
I20240502 21:35:47.154933 26697 raft_consensus.cc:2932] T d5cf433249bc4c309c56c9d2e02efebe P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5b1043ab39c04568814296d029600092 (127.27.196.66) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { promote: false } } }
I20240502 21:35:47.155923 26282 catalog_manager.cc:5484] T d5cf433249bc4c309c56c9d2e02efebe P 27be1297e75b4e20ba297e7332a98f92 reported cstate change: config changed from index 7 to 8, 5b1043ab39c04568814296d029600092 (127.27.196.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cb27977c77564d38afe6e64c816eacd0" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { replace: true } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { promote: false } } }
I20240502 21:35:47.170063 26430 consensus_queue.cc:237] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { promote: false } }
I20240502 21:35:47.173667 26564 raft_consensus.cc:1261] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:35:47.174377 28628 consensus_queue.cc:1035] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:35:47.177294 28653 tablet_replica.cc:330] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc: stopping tablet replica
I20240502 21:35:47.177603 28653 raft_consensus.cc:2229] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:35:47.177850 27510 raft_consensus.cc:1261] T d5cf433249bc4c309c56c9d2e02efebe P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:35:47.178328 28653 raft_consensus.cc:2258] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:35:47.178334 28560 raft_consensus.cc:2932] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 63266faf827046629e39afa765b144bc (127.27.196.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { promote: false } } }
I20240502 21:35:47.179328 27925 consensus_queue.cc:1035] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:35:47.180253 26565 raft_consensus.cc:2932] T d5cf433249bc4c309c56c9d2e02efebe P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 63266faf827046629e39afa765b144bc (127.27.196.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { promote: false } } }
I20240502 21:35:47.181066 28653 ts_tablet_manager.cc:1875] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:35:47.183199 26273 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d5cf433249bc4c309c56c9d2e02efebe with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:35:47.184655 27510 raft_consensus.cc:2932] T d5cf433249bc4c309c56c9d2e02efebe P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 63266faf827046629e39afa765b144bc (127.27.196.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { promote: false } } }
I20240502 21:35:47.185052 26284 catalog_manager.cc:5484] T d5cf433249bc4c309c56c9d2e02efebe P cb27977c77564d38afe6e64c816eacd0 reported cstate change: config changed from index 8 to 9, VOTER 63266faf827046629e39afa765b144bc (127.27.196.67) evicted. New cstate: current_term: 1 leader_uuid: "cb27977c77564d38afe6e64c816eacd0" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:35:47.192526 26678 tablet_service.cc:1514] Processing DeleteTablet for tablet d5cf433249bc4c309c56c9d2e02efebe with delete_type TABLET_DATA_TOMBSTONED (TS 63266faf827046629e39afa765b144bc not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:42438
I20240502 21:35:47.200136 28653 ts_tablet_manager.cc:1888] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:35:47.200385 28653 log.cc:1196] T 8747e6794c084510a5b9e1f34c15c733 P 63266faf827046629e39afa765b144bc: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/wals/8747e6794c084510a5b9e1f34c15c733
I20240502 21:35:47.201599 28653 tablet_replica.cc:330] T d5cf433249bc4c309c56c9d2e02efebe P 63266faf827046629e39afa765b144bc: stopping tablet replica
I20240502 21:35:47.201577 26274 catalog_manager.cc:4830] TS 63266faf827046629e39afa765b144bc (127.27.196.67:39203): tablet 8747e6794c084510a5b9e1f34c15c733 (table rebalance_test_table_0 [id=7277b1f7f8b747f4a8e65521bf4af6c0]) successfully deleted
I20240502 21:35:47.202064 28653 raft_consensus.cc:2229] T d5cf433249bc4c309c56c9d2e02efebe P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:35:47.202428 28653 raft_consensus.cc:2258] T d5cf433249bc4c309c56c9d2e02efebe P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:35:47.203792 28653 ts_tablet_manager.cc:1875] T d5cf433249bc4c309c56c9d2e02efebe P 63266faf827046629e39afa765b144bc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:35:47.210604 28653 ts_tablet_manager.cc:1888] T d5cf433249bc4c309c56c9d2e02efebe P 63266faf827046629e39afa765b144bc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:35:47.210793 28653 log.cc:1196] T d5cf433249bc4c309c56c9d2e02efebe P 63266faf827046629e39afa765b144bc: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/wals/d5cf433249bc4c309c56c9d2e02efebe
I20240502 21:35:47.211889 26274 catalog_manager.cc:4830] TS 63266faf827046629e39afa765b144bc (127.27.196.67:39203): tablet d5cf433249bc4c309c56c9d2e02efebe (table rebalance_test_table_2 [id=79000138ee6c4b96965a29a347143ffa]) successfully deleted
I20240502 21:35:47.223169 28628 raft_consensus.cc:1050] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0: attempting to promote NON_VOTER 63266faf827046629e39afa765b144bc to VOTER
I20240502 21:35:47.224300 28628 consensus_queue.cc:237] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false } }
I20240502 21:35:47.226960 27510 raft_consensus.cc:1261] T 140f7280768e4ad592309e9b7d6a26da P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:35:47.227371 26564 raft_consensus.cc:1261] T 140f7280768e4ad592309e9b7d6a26da P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:35:47.227741 26697 raft_consensus.cc:1261] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc [term 1 LEARNER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:35:47.227813 27925 consensus_queue.cc:1035] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:35:47.228827 28561 consensus_queue.cc:1035] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:35:47.229396 28561 consensus_queue.cc:1035] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:35:47.244386 28628 raft_consensus.cc:2932] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 63266faf827046629e39afa765b144bc (127.27.196.67) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false } } }
I20240502 21:35:47.244535 26564 raft_consensus.cc:2932] T 140f7280768e4ad592309e9b7d6a26da P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 63266faf827046629e39afa765b144bc (127.27.196.67) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false } } }
I20240502 21:35:47.246713 26697 raft_consensus.cc:2932] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 63266faf827046629e39afa765b144bc (127.27.196.67) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false } } }
I20240502 21:35:47.249722 27510 raft_consensus.cc:2932] T 140f7280768e4ad592309e9b7d6a26da P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 63266faf827046629e39afa765b144bc (127.27.196.67) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false } } }
I20240502 21:35:47.251399 26284 catalog_manager.cc:5484] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 reported cstate change: config changed from index 7 to 8, 63266faf827046629e39afa765b144bc (127.27.196.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cb27977c77564d38afe6e64c816eacd0" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:35:47.259419 26430 consensus_queue.cc:237] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false } }
I20240502 21:35:47.261853 26697 raft_consensus.cc:1261] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:35:47.262136 27510 raft_consensus.cc:1261] T 140f7280768e4ad592309e9b7d6a26da P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Refusing update from remote peer cb27977c77564d38afe6e64c816eacd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:35:47.262626 28628 consensus_queue.cc:1035] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:35:47.263064 28561 consensus_queue.cc:1035] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:35:47.265869 28561 raft_consensus.cc:2932] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5b1043ab39c04568814296d029600092 (127.27.196.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false } } }
I20240502 21:35:47.266584 27510 raft_consensus.cc:2932] T 140f7280768e4ad592309e9b7d6a26da P 27be1297e75b4e20ba297e7332a98f92 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5b1043ab39c04568814296d029600092 (127.27.196.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false } } }
I20240502 21:35:47.268318 26697 raft_consensus.cc:2932] T 140f7280768e4ad592309e9b7d6a26da P 63266faf827046629e39afa765b144bc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5b1043ab39c04568814296d029600092 (127.27.196.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false } } }
I20240502 21:35:47.269871 26273 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 140f7280768e4ad592309e9b7d6a26da with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:35:47.271828 26286 catalog_manager.cc:5484] T 140f7280768e4ad592309e9b7d6a26da P cb27977c77564d38afe6e64c816eacd0 reported cstate change: config changed from index 8 to 9, VOTER 5b1043ab39c04568814296d029600092 (127.27.196.66) evicted. New cstate: current_term: 1 leader_uuid: "cb27977c77564d38afe6e64c816eacd0" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:35:47.276983 26543 tablet_service.cc:1514] Processing DeleteTablet for tablet 140f7280768e4ad592309e9b7d6a26da with delete_type TABLET_DATA_TOMBSTONED (TS 5b1043ab39c04568814296d029600092 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:34904
I20240502 21:35:47.278177 28675 tablet_replica.cc:330] T 140f7280768e4ad592309e9b7d6a26da P 5b1043ab39c04568814296d029600092: stopping tablet replica
I20240502 21:35:47.278474 28675 raft_consensus.cc:2229] T 140f7280768e4ad592309e9b7d6a26da P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:35:47.279580 28675 raft_consensus.cc:2258] T 140f7280768e4ad592309e9b7d6a26da P 5b1043ab39c04568814296d029600092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:35:47.281188 28675 ts_tablet_manager.cc:1875] T 140f7280768e4ad592309e9b7d6a26da P 5b1043ab39c04568814296d029600092: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:35:47.290480 28675 ts_tablet_manager.cc:1888] T 140f7280768e4ad592309e9b7d6a26da P 5b1043ab39c04568814296d029600092: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:35:47.290674 28675 log.cc:1196] T 140f7280768e4ad592309e9b7d6a26da P 5b1043ab39c04568814296d029600092: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/wals/140f7280768e4ad592309e9b7d6a26da
I20240502 21:35:47.291771 26273 catalog_manager.cc:4830] TS 5b1043ab39c04568814296d029600092 (127.27.196.66:41285): tablet 140f7280768e4ad592309e9b7d6a26da (table rebalance_test_table_1 [id=cc5055b898c74d0ea2d82c8d8f1f5bb4]) successfully deleted
I20240502 21:35:47.298957 26430 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "35508ded8421457288bdef00bf12f698"
dest_uuid: "cb27977c77564d38afe6e64c816eacd0"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:37580
I20240502 21:35:47.299300 26430 raft_consensus.cc:595] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 2 LEADER]: Received request to transfer leadership
I20240502 21:35:47.345571 26430 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "35508ded8421457288bdef00bf12f698"
dest_uuid: "cb27977c77564d38afe6e64c816eacd0"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:37600
I20240502 21:35:47.345832 26430 raft_consensus.cc:595] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 2 LEADER]: Received request to transfer leadership
I20240502 21:35:47.368026 26430 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "35508ded8421457288bdef00bf12f698"
dest_uuid: "cb27977c77564d38afe6e64c816eacd0"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:37622
I20240502 21:35:47.368283 26430 raft_consensus.cc:595] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 2 LEADER]: Received request to transfer leadership
I20240502 21:35:47.377470 28628 raft_consensus.cc:979] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0: : Instructing follower 63266faf827046629e39afa765b144bc to start an election
I20240502 21:35:47.377764 28628 raft_consensus.cc:1067] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 2 LEADER]: Signalling peer 63266faf827046629e39afa765b144bc to start an election
I20240502 21:35:47.378983 26697 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "35508ded8421457288bdef00bf12f698"
dest_uuid: "63266faf827046629e39afa765b144bc"
 from {username='slave'} at 127.27.196.65:54089
I20240502 21:35:47.379276 26697 raft_consensus.cc:482] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:35:47.379446 26697 raft_consensus.cc:3037] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:35:47.382688 26697 raft_consensus.cc:504] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false } }
I20240502 21:35:47.383702 26697 leader_election.cc:288] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [CANDIDATE]: Term 3 election: Requested vote from peers cb27977c77564d38afe6e64c816eacd0 (127.27.196.65:36209), 5b1043ab39c04568814296d029600092 (127.27.196.66:41285), 27be1297e75b4e20ba297e7332a98f92 (127.27.196.68:35953)
I20240502 21:35:47.384302 26430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35508ded8421457288bdef00bf12f698" candidate_uuid: "63266faf827046629e39afa765b144bc" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "cb27977c77564d38afe6e64c816eacd0"
I20240502 21:35:47.384430 26564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35508ded8421457288bdef00bf12f698" candidate_uuid: "63266faf827046629e39afa765b144bc" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "5b1043ab39c04568814296d029600092"
I20240502 21:35:47.384559 26430 raft_consensus.cc:3032] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:35:47.384687 26430 raft_consensus.cc:729] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cb27977c77564d38afe6e64c816eacd0, State: Running, Role: LEADER
I20240502 21:35:47.384685 26564 raft_consensus.cc:3037] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:35:47.384773 27510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35508ded8421457288bdef00bf12f698" candidate_uuid: "63266faf827046629e39afa765b144bc" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "27be1297e75b4e20ba297e7332a98f92"
I20240502 21:35:47.385005 27510 raft_consensus.cc:3037] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:35:47.385103 26430 consensus_queue.cc:260] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false } }
I20240502 21:35:47.385627 26430 raft_consensus.cc:3037] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:35:47.388775 27510 raft_consensus.cc:2454] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63266faf827046629e39afa765b144bc in term 3.
I20240502 21:35:47.388784 26564 raft_consensus.cc:2454] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63266faf827046629e39afa765b144bc in term 3.
I20240502 21:35:47.389196 26430 raft_consensus.cc:2454] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63266faf827046629e39afa765b144bc in term 3.
I20240502 21:35:47.389592 26632 leader_election.cc:302] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 27be1297e75b4e20ba297e7332a98f92, 5b1043ab39c04568814296d029600092, 63266faf827046629e39afa765b144bc; no voters: 
I20240502 21:35:47.389961 28336 raft_consensus.cc:2781] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:35:47.390453 28336 raft_consensus.cc:686] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 3 LEADER]: Becoming Leader. State: Replica: 63266faf827046629e39afa765b144bc, State: Running, Role: LEADER
I20240502 21:35:47.390785 28336 consensus_queue.cc:237] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false } }
I20240502 21:35:47.393901 26282 catalog_manager.cc:5484] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc reported cstate change: term changed from 2 to 3, leader changed from cb27977c77564d38afe6e64c816eacd0 (127.27.196.65) to 63266faf827046629e39afa765b144bc (127.27.196.67). New cstate: current_term: 3 leader_uuid: "63266faf827046629e39afa765b144bc" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:35:47.808935 27510 raft_consensus.cc:1261] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92 [term 3 FOLLOWER]: Refusing update from remote peer 63266faf827046629e39afa765b144bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 21:35:47.809775 28336 consensus_queue.cc:1035] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:35:47.818722 26564 raft_consensus.cc:1261] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092 [term 3 FOLLOWER]: Refusing update from remote peer 63266faf827046629e39afa765b144bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 21:35:47.819394 28336 consensus_queue.cc:1035] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:35:47.824621 26430 raft_consensus.cc:1261] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 3 FOLLOWER]: Refusing update from remote peer 63266faf827046629e39afa765b144bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 21:35:47.827379 28336 consensus_queue.cc:1035] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb27977c77564d38afe6e64c816eacd0" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 36209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:35:47.828619 26697 consensus_queue.cc:237] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false } }
I20240502 21:35:47.831619 27510 raft_consensus.cc:1261] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92 [term 3 FOLLOWER]: Refusing update from remote peer 63266faf827046629e39afa765b144bc: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 21:35:47.831998 26564 raft_consensus.cc:1261] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092 [term 3 FOLLOWER]: Refusing update from remote peer 63266faf827046629e39afa765b144bc: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 21:35:47.832405 28336 consensus_queue.cc:1035] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:35:47.832940 28711 consensus_queue.cc:1035] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:35:47.835636 28712 raft_consensus.cc:2932] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER cb27977c77564d38afe6e64c816eacd0 (127.27.196.65) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false } } }
I20240502 21:35:47.836222 27510 raft_consensus.cc:2932] T 35508ded8421457288bdef00bf12f698 P 27be1297e75b4e20ba297e7332a98f92 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER cb27977c77564d38afe6e64c816eacd0 (127.27.196.65) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false } } }
I20240502 21:35:47.840754 26274 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 35508ded8421457288bdef00bf12f698 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:35:47.842139 26564 raft_consensus.cc:2932] T 35508ded8421457288bdef00bf12f698 P 5b1043ab39c04568814296d029600092 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER cb27977c77564d38afe6e64c816eacd0 (127.27.196.65) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false } } }
I20240502 21:35:47.842204 26286 catalog_manager.cc:5484] T 35508ded8421457288bdef00bf12f698 P 63266faf827046629e39afa765b144bc reported cstate change: config changed from index 9 to 11, VOTER cb27977c77564d38afe6e64c816eacd0 (127.27.196.65) evicted. New cstate: current_term: 3 leader_uuid: "63266faf827046629e39afa765b144bc" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5b1043ab39c04568814296d029600092" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 41285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27be1297e75b4e20ba297e7332a98f92" member_type: VOTER last_known_addr { host: "127.27.196.68" port: 35953 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63266faf827046629e39afa765b144bc" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 39203 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:35:47.847669 26409 tablet_service.cc:1514] Processing DeleteTablet for tablet 35508ded8421457288bdef00bf12f698 with delete_type TABLET_DATA_TOMBSTONED (TS cb27977c77564d38afe6e64c816eacd0 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:33746
I20240502 21:35:47.849154 28731 tablet_replica.cc:330] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0: stopping tablet replica
I20240502 21:35:47.849395 28731 raft_consensus.cc:2229] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:35:47.849622 28731 raft_consensus.cc:2258] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:35:47.850672 28731 ts_tablet_manager.cc:1875] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:35:47.855432 28731 ts_tablet_manager.cc:1888] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.10
I20240502 21:35:47.855583 28731 log.cc:1196] T 35508ded8421457288bdef00bf12f698 P cb27977c77564d38afe6e64c816eacd0: Deleting WAL directory at /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/wals/35508ded8421457288bdef00bf12f698
I20240502 21:35:47.856627 26273 catalog_manager.cc:4830] TS cb27977c77564d38afe6e64c816eacd0 (127.27.196.65:36209): tablet 35508ded8421457288bdef00bf12f698 (table rebalance_test_table_1 [id=cc5055b898c74d0ea2d82c8d8f1f5bb4]) successfully deleted
I20240502 21:35:48.129562 26543 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:48.130731 27490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:35:48.133090 26678 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:35:48.149410 26409 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:35:48.165904 27354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:35:48.339186 27490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:35:48.343685 26678 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:35:48.353494 26543 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:35:48.360563 27354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:35:48.368381 26409 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:35:48.611251 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 26347
I20240502 21:35:48.631866 26470 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:35:48.823144 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 26347
I20240502 21:35:48.855615 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 26480
I20240502 21:35:48.862426 26605 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:35:49.012404 26632 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:36209: connect: Connection refused (error 111)
W20240502 21:35:49.016435 26632 consensus_peers.cc:485] T e5bca5fdd4d14b5baaa0b0cfc4754a23 P 63266faf827046629e39afa765b144bc -> Peer cb27977c77564d38afe6e64c816eacd0 (127.27.196.65:36209): Couldn't send request to peer cb27977c77564d38afe6e64c816eacd0. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:36209: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:35:49.029417 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 26480
I20240502 21:35:49.062028 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 26615
I20240502 21:35:49.069056 26739 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:35:49.223413 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 26615
W20240502 21:35:49.263417 27441 connection.cc:535] server connection from 127.27.196.67:42929 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:35:49.264393 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 27425
I20240502 21:35:49.272280 27551 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:35:49.413072 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 27425
I20240502 21:35:49.441954 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 27291
I20240502 21:35:49.449764 27415 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:35:49.572459 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 27291
I20240502 21:35:49.598313 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 26256
I20240502 21:35:49.599550 26318 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:35:49.757781 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 26256
2024-05-02T21:35:49Z chronyd exiting
[       OK ] LocationAwareRebalancingParamTest.Rebalance/idx2_rf3_t3_l_3_2 (24032 ms)
[ RUN      ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx1_rf3_t3_l_3_1
I20240502 21:35:49.821938 28433 test_util.cc:268] Using random seed: 1661272534
I20240502 21:35:49.824091 28433 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:35:49.824168 28433 ts_itest-base.cc:113] --------------
I20240502 21:35:49.824235 28433 ts_itest-base.cc:114] 4 tablet servers
I20240502 21:35:49.824285 28433 ts_itest-base.cc:115] 3 replicas per TS
I20240502 21:35:49.824357 28433 ts_itest-base.cc:116] --------------
2024-05-02T21:35:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:35:49Z Disabled control of system clock
I20240502 21:35:49.902397 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.196.126:46355
--webserver_interface=127.27.196.126
--webserver_port=0
--builtin_ntp_servers=127.27.196.84:33157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.196.126:46355
--location_mapping_cmd=/tmp/dist-test-taskBPlvDh/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:1
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240502 21:35:50.280241 28823 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 21:35:50.280656 28823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:35:50.280802 28823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:35:50.280922 28823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:35:50.288847 28823 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:35:50.289033 28823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:35:50.289146 28823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:35:50.289213 28823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:35:50.289307 28823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:35:50.299326 28823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:33157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskBPlvDh/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:1
--ipki_ca_key_size=768
--master_addresses=127.27.196.126:46355
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.196.126:46355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.27.196.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:35:50.300812 28823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:35:50.302665 28823 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:35:50.314817 28828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:50.314814 28829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:50.315253 28831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:35:50.315827 28823 server_base.cc:1034] running on GCE node
I20240502 21:35:50.316725 28823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:35:50.318403 28823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:35:50.319653 28823 hybrid_clock.cc:648] HybridClock initialized: now 1714685750319567 us; error 61 us; skew 500 ppm
I20240502 21:35:50.320016 28823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:35:50.328238 28823 webserver.cc:460] Webserver started at http://127.27.196.126:42563/ using document root <none> and password file <none>
I20240502 21:35:50.329144 28823 fs_manager.cc:359] Metadata directory not provided
I20240502 21:35:50.329284 28823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:35:50.329631 28823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:35:50.332677 28823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "6d2a86d40db54a0c9f8ab6a5d45ba851"
format_stamp: "Formatted at 2024-05-02 21:35:50 on dist-test-slave-3bs1"
I20240502 21:35:50.333534 28823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "6d2a86d40db54a0c9f8ab6a5d45ba851"
format_stamp: "Formatted at 2024-05-02 21:35:50 on dist-test-slave-3bs1"
I20240502 21:35:50.339282 28823 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.004s
I20240502 21:35:50.343071 28837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:50.344683 28823 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 21:35:50.344892 28823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
uuid: "6d2a86d40db54a0c9f8ab6a5d45ba851"
format_stamp: "Formatted at 2024-05-02 21:35:50 on dist-test-slave-3bs1"
I20240502 21:35:50.345104 28823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:35:50.366860 28823 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:35:50.367883 28823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:35:50.368201 28823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:35:50.389968 28823 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.126:46355
I20240502 21:35:50.390002 28891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.126:46355 every 8 connection(s)
I20240502 21:35:50.391664 28823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 21:35:50.393301 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 28823
I20240502 21:35:50.393550 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:35:50.397028 28892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:50.409739 28892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d2a86d40db54a0c9f8ab6a5d45ba851: Bootstrap starting.
I20240502 21:35:50.413463 28892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d2a86d40db54a0c9f8ab6a5d45ba851: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:50.414534 28892 log.cc:824] T 00000000000000000000000000000000 P 6d2a86d40db54a0c9f8ab6a5d45ba851: Log is configured to *not* fsync() on all Append() calls
I20240502 21:35:50.417416 28892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d2a86d40db54a0c9f8ab6a5d45ba851: No bootstrap required, opened a new log
I20240502 21:35:50.424293 28892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d2a86d40db54a0c9f8ab6a5d45ba851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2a86d40db54a0c9f8ab6a5d45ba851" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 46355 } }
I20240502 21:35:50.424705 28892 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d2a86d40db54a0c9f8ab6a5d45ba851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:50.424811 28892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d2a86d40db54a0c9f8ab6a5d45ba851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d2a86d40db54a0c9f8ab6a5d45ba851, State: Initialized, Role: FOLLOWER
I20240502 21:35:50.425451 28892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d2a86d40db54a0c9f8ab6a5d45ba851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2a86d40db54a0c9f8ab6a5d45ba851" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 46355 } }
I20240502 21:35:50.425675 28892 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d2a86d40db54a0c9f8ab6a5d45ba851 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:35:50.425823 28892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d2a86d40db54a0c9f8ab6a5d45ba851 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:35:50.425994 28892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d2a86d40db54a0c9f8ab6a5d45ba851 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:50.428344 28892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d2a86d40db54a0c9f8ab6a5d45ba851 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2a86d40db54a0c9f8ab6a5d45ba851" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 46355 } }
I20240502 21:35:50.428928 28892 leader_election.cc:302] T 00000000000000000000000000000000 P 6d2a86d40db54a0c9f8ab6a5d45ba851 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d2a86d40db54a0c9f8ab6a5d45ba851; no voters: 
I20240502 21:35:50.429616 28892 leader_election.cc:288] T 00000000000000000000000000000000 P 6d2a86d40db54a0c9f8ab6a5d45ba851 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 21:35:50.429796 28897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d2a86d40db54a0c9f8ab6a5d45ba851 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:50.430598 28897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d2a86d40db54a0c9f8ab6a5d45ba851 [term 1 LEADER]: Becoming Leader. State: Replica: 6d2a86d40db54a0c9f8ab6a5d45ba851, State: Running, Role: LEADER
I20240502 21:35:50.431372 28897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d2a86d40db54a0c9f8ab6a5d45ba851 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2a86d40db54a0c9f8ab6a5d45ba851" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 46355 } }
I20240502 21:35:50.432161 28892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d2a86d40db54a0c9f8ab6a5d45ba851 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:35:50.434938 28899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d2a86d40db54a0c9f8ab6a5d45ba851 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d2a86d40db54a0c9f8ab6a5d45ba851. Latest consensus state: current_term: 1 leader_uuid: "6d2a86d40db54a0c9f8ab6a5d45ba851" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2a86d40db54a0c9f8ab6a5d45ba851" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 46355 } } }
I20240502 21:35:50.435537 28899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d2a86d40db54a0c9f8ab6a5d45ba851 [sys.catalog]: This master's current role is: LEADER
I20240502 21:35:50.435832 28898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d2a86d40db54a0c9f8ab6a5d45ba851 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d2a86d40db54a0c9f8ab6a5d45ba851" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2a86d40db54a0c9f8ab6a5d45ba851" member_type: VOTER last_known_addr { host: "127.27.196.126" port: 46355 } } }
I20240502 21:35:50.436169 28898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d2a86d40db54a0c9f8ab6a5d45ba851 [sys.catalog]: This master's current role is: LEADER
I20240502 21:35:50.437731 28903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:35:50.443809 28903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:35:50.455873 28903 catalog_manager.cc:1347] Generated new cluster ID: 4783b77412054e50afee8c9ee12cf1c1
I20240502 21:35:50.456023 28903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:35:50.476186 28903 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:35:50.478714 28903 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:35:50.502825 28903 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6d2a86d40db54a0c9f8ab6a5d45ba851: Generated new TSK 0
I20240502 21:35:50.504120 28903 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:35:50.521399 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.65:0
--local_ip_for_outbound_sockets=127.27.196.65
--webserver_interface=127.27.196.65
--webserver_port=0
--tserver_master_addrs=127.27.196.126:46355
--builtin_ntp_servers=127.27.196.84:33157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:35:50.933218 28916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:35:50.933584 28916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:35:50.933732 28916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:35:50.942047 28916 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:35:50.942185 28916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:35:50.942346 28916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.65
I20240502 21:35:50.952750 28916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:33157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.27.196.65
--webserver_port=0
--tserver_master_addrs=127.27.196.126:46355
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.65
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:35:50.954270 28916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:35:50.956224 28916 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:35:50.969259 28922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:50.969830 28923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:50.970326 28925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:35:50.972844 28916 server_base.cc:1034] running on GCE node
I20240502 21:35:50.973579 28916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:35:50.974663 28916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:35:50.975914 28916 hybrid_clock.cc:648] HybridClock initialized: now 1714685750975854 us; error 48 us; skew 500 ppm
I20240502 21:35:50.976356 28916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:35:50.980687 28916 webserver.cc:460] Webserver started at http://127.27.196.65:33599/ using document root <none> and password file <none>
I20240502 21:35:50.981832 28916 fs_manager.cc:359] Metadata directory not provided
I20240502 21:35:50.981993 28916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:35:50.982456 28916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:35:50.987391 28916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2"
format_stamp: "Formatted at 2024-05-02 21:35:50 on dist-test-slave-3bs1"
I20240502 21:35:50.988380 28916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2"
format_stamp: "Formatted at 2024-05-02 21:35:50 on dist-test-slave-3bs1"
I20240502 21:35:50.994371 28916 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.001s
I20240502 21:35:50.998482 28931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:51.000023 28916 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
I20240502 21:35:51.000190 28916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2"
format_stamp: "Formatted at 2024-05-02 21:35:50 on dist-test-slave-3bs1"
I20240502 21:35:51.000404 28916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:35:51.025306 28916 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:35:51.026239 28916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:35:51.026847 28916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:35:51.028542 28916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:35:51.030438 28916 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:35:51.030544 28916 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:51.030654 28916 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:35:51.030711 28916 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:51.069914 28916 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.65:42025
I20240502 21:35:51.069970 29048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.65:42025 every 8 connection(s)
I20240502 21:35:51.071467 28916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:35:51.076256 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 28916
I20240502 21:35:51.076525 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:35:51.081578 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.66:0
--local_ip_for_outbound_sockets=127.27.196.66
--webserver_interface=127.27.196.66
--webserver_port=0
--tserver_master_addrs=127.27.196.126:46355
--builtin_ntp_servers=127.27.196.84:33157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:35:51.091399 29049 heartbeater.cc:347] Connected to a master server at 127.27.196.126:46355
I20240502 21:35:51.091989 29049 heartbeater.cc:464] Registering TS with master...
I20240502 21:35:51.093578 29049 heartbeater.cc:511] Master 127.27.196.126:46355 requested a full tablet report, sending...
I20240502 21:35:51.174430 28855 ts_manager.cc:194] Registered new tserver with Master: 5da7dc5c70ff4f3fa6abf7cc75bb12f2 (127.27.196.65:42025)
I20240502 21:35:51.177150 28855 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.65:60551
W20240502 21:35:51.539319 29053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:35:51.539656 29053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:35:51.539760 29053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:35:51.548133 29053 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:35:51.548292 29053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:35:51.548430 29053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.66
I20240502 21:35:51.559720 29053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:33157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.27.196.66
--webserver_port=0
--tserver_master_addrs=127.27.196.126:46355
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.66
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:35:51.561170 29053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:35:51.563041 29053 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:35:51.576848 29060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:51.577865 29059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:51.578071 29062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:35:51.578655 29053 server_base.cc:1034] running on GCE node
I20240502 21:35:51.579552 29053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:35:51.580600 29053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:35:51.581825 29053 hybrid_clock.cc:648] HybridClock initialized: now 1714685751581758 us; error 56 us; skew 500 ppm
I20240502 21:35:51.582165 29053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:35:51.585289 29053 webserver.cc:460] Webserver started at http://127.27.196.66:39613/ using document root <none> and password file <none>
I20240502 21:35:51.586331 29053 fs_manager.cc:359] Metadata directory not provided
I20240502 21:35:51.586459 29053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:35:51.586872 29053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:35:51.591106 29053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "8148636210184b38b1fe012cd2d0eb29"
format_stamp: "Formatted at 2024-05-02 21:35:51 on dist-test-slave-3bs1"
I20240502 21:35:51.592159 29053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "8148636210184b38b1fe012cd2d0eb29"
format_stamp: "Formatted at 2024-05-02 21:35:51 on dist-test-slave-3bs1"
I20240502 21:35:51.598553 29053 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.000s
I20240502 21:35:51.602852 29068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:51.604820 29053 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 21:35:51.605044 29053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "8148636210184b38b1fe012cd2d0eb29"
format_stamp: "Formatted at 2024-05-02 21:35:51 on dist-test-slave-3bs1"
I20240502 21:35:51.605283 29053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:35:51.623576 29053 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:35:51.624683 29053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:35:51.624970 29053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:35:51.626399 29053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:35:51.628868 29053 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:35:51.629001 29053 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:51.629104 29053 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:35:51.629164 29053 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:51.672894 29053 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.66:38577
I20240502 21:35:51.672976 29184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.66:38577 every 8 connection(s)
I20240502 21:35:51.674377 29053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:35:51.676045 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 29053
I20240502 21:35:51.676254 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:35:51.680173 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.67:0
--local_ip_for_outbound_sockets=127.27.196.67
--webserver_interface=127.27.196.67
--webserver_port=0
--tserver_master_addrs=127.27.196.126:46355
--builtin_ntp_servers=127.27.196.84:33157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:35:51.692368 29185 heartbeater.cc:347] Connected to a master server at 127.27.196.126:46355
I20240502 21:35:51.692984 29185 heartbeater.cc:464] Registering TS with master...
I20240502 21:35:51.694236 29185 heartbeater.cc:511] Master 127.27.196.126:46355 requested a full tablet report, sending...
I20240502 21:35:51.749758 28855 ts_manager.cc:194] Registered new tserver with Master: 8148636210184b38b1fe012cd2d0eb29 (127.27.196.66:38577)
I20240502 21:35:51.751380 28855 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.66:49161
W20240502 21:35:52.088868 29189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:35:52.089212 29189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:35:52.089311 29189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:35:52.097410 29189 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:35:52.097553 29189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:35:52.097724 29189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.67
I20240502 21:35:52.108147 29189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:33157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.27.196.67
--webserver_port=0
--tserver_master_addrs=127.27.196.126:46355
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.67
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:35:52.110726 29189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:35:52.113301 29189 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:35:52.130336 29198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:52.131139 29196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:52.131474 29195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:35:52.132165 29189 server_base.cc:1034] running on GCE node
I20240502 21:35:52.132987 29189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:35:52.134236 29189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:35:52.135488 29189 hybrid_clock.cc:648] HybridClock initialized: now 1714685752135431 us; error 49 us; skew 500 ppm
I20240502 21:35:52.135874 29189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:35:52.139663 29189 webserver.cc:460] Webserver started at http://127.27.196.67:42647/ using document root <none> and password file <none>
I20240502 21:35:52.140960 29189 fs_manager.cc:359] Metadata directory not provided
I20240502 21:35:52.141171 29189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:35:52.141661 29189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:35:52.145205 29189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba"
format_stamp: "Formatted at 2024-05-02 21:35:52 on dist-test-slave-3bs1"
I20240502 21:35:52.146148 29189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba"
format_stamp: "Formatted at 2024-05-02 21:35:52 on dist-test-slave-3bs1"
I20240502 21:35:52.152530 29189 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.002s
I20240502 21:35:52.156884 29204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:52.158424 29189 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.002s
I20240502 21:35:52.158625 29189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba"
format_stamp: "Formatted at 2024-05-02 21:35:52 on dist-test-slave-3bs1"
I20240502 21:35:52.158871 29189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:35:52.177536 29189 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:35:52.178520 29189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:35:52.178823 29189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:35:52.179981 29189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:35:52.181718 29189 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:35:52.181706 29049 heartbeater.cc:503] Master 127.27.196.126:46355 was elected leader, sending a full tablet report...
I20240502 21:35:52.181838 29189 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:52.181964 29189 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:35:52.182050 29189 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:52.218734 29189 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.67:36675
I20240502 21:35:52.218781 29320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.67:36675 every 8 connection(s)
I20240502 21:35:52.220175 29189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:35:52.225950 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 29189
I20240502 21:35:52.226205 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:35:52.230185 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.68:0
--local_ip_for_outbound_sockets=127.27.196.68
--webserver_interface=127.27.196.68
--webserver_port=0
--tserver_master_addrs=127.27.196.126:46355
--builtin_ntp_servers=127.27.196.84:33157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:35:52.236582 29321 heartbeater.cc:347] Connected to a master server at 127.27.196.126:46355
I20240502 21:35:52.236958 29321 heartbeater.cc:464] Registering TS with master...
I20240502 21:35:52.237977 29321 heartbeater.cc:511] Master 127.27.196.126:46355 requested a full tablet report, sending...
I20240502 21:35:52.304181 28855 ts_manager.cc:194] Registered new tserver with Master: a3ecdc3dc21946cfb5f4a2e5ab0a75ba (127.27.196.67:36675)
I20240502 21:35:52.305579 28855 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.67:46171
W20240502 21:35:52.646133 29325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:35:52.646463 29325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:35:52.646593 29325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:35:52.659235 29325 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:35:52.659421 29325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:35:52.659629 29325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.68
I20240502 21:35:52.673017 29325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:33157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.27.196.68
--webserver_port=0
--tserver_master_addrs=127.27.196.126:46355
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.68
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:35:52.674420 29325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:35:52.676275 29325 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:35:52.690882 29331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:52.691430 29332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:35:52.691958 29334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:35:52.692437 29325 server_base.cc:1034] running on GCE node
I20240502 21:35:52.693149 29325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:35:52.694360 29325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:35:52.695641 29325 hybrid_clock.cc:648] HybridClock initialized: now 1714685752695585 us; error 48 us; skew 500 ppm
I20240502 21:35:52.696015 29325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:35:52.698902 29325 webserver.cc:460] Webserver started at http://127.27.196.68:38177/ using document root <none> and password file <none>
I20240502 21:35:52.699843 29325 fs_manager.cc:359] Metadata directory not provided
I20240502 21:35:52.699980 29325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:35:52.700312 29325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:35:52.703186 29325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "05a47f25c377450b930558c362256996"
format_stamp: "Formatted at 2024-05-02 21:35:52 on dist-test-slave-3bs1"
I20240502 21:35:52.704010 29325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "05a47f25c377450b930558c362256996"
format_stamp: "Formatted at 2024-05-02 21:35:52 on dist-test-slave-3bs1"
I20240502 21:35:52.709988 29325 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20240502 21:35:52.713979 29340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:52.715603 29325 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 21:35:52.715787 29325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "05a47f25c377450b930558c362256996"
format_stamp: "Formatted at 2024-05-02 21:35:52 on dist-test-slave-3bs1"
I20240502 21:35:52.716027 29325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:35:52.734125 29325 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:35:52.735086 29325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:35:52.735435 29325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:35:52.736527 29325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:35:52.738197 29325 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:35:52.738308 29325 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:52.738427 29325 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:35:52.738502 29325 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:35:52.755553 29185 heartbeater.cc:503] Master 127.27.196.126:46355 was elected leader, sending a full tablet report...
I20240502 21:35:52.775565 29325 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.68:39161
I20240502 21:35:52.775621 29454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.68:39161 every 8 connection(s)
I20240502 21:35:52.777051 29325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:35:52.781903 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 29325
I20240502 21:35:52.782094 28433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 21:35:52.792567 29455 heartbeater.cc:347] Connected to a master server at 127.27.196.126:46355
I20240502 21:35:52.792920 29455 heartbeater.cc:464] Registering TS with master...
I20240502 21:35:52.793884 29455 heartbeater.cc:511] Master 127.27.196.126:46355 requested a full tablet report, sending...
I20240502 21:35:52.843959 28855 ts_manager.cc:194] Registered new tserver with Master: 05a47f25c377450b930558c362256996 (127.27.196.68:39161)
I20240502 21:35:52.844945 28855 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.68:46681
I20240502 21:35:52.856036 28433 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 21:35:52.875077 28433 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 21:35:52.875281 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 29325
I20240502 21:35:52.884510 29448 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:35:52.969794 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 29325
I20240502 21:35:53.309636 29321 heartbeater.cc:503] Master 127.27.196.126:46355 was elected leader, sending a full tablet report...
I20240502 21:35:56.746903 28855 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59708:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240502 21:35:56.754463 28855 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 21:35:56.796515 29116 tablet_service.cc:1467] Processing CreateTablet for tablet b44bc823d03c42b690b1e3fef7371e7b (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07faf32dc6e4d7880b99f6f9d18c408]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:56.798270 29116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44bc823d03c42b690b1e3fef7371e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:56.798334 28980 tablet_service.cc:1467] Processing CreateTablet for tablet b44bc823d03c42b690b1e3fef7371e7b (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07faf32dc6e4d7880b99f6f9d18c408]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:56.798673 28979 tablet_service.cc:1467] Processing CreateTablet for tablet 87f738af6dec4151b08895d4f753fc5e (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07faf32dc6e4d7880b99f6f9d18c408]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:56.797797 28981 tablet_service.cc:1467] Processing CreateTablet for tablet f6c1142c68564c98b6c942ffd8c0efdc (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07faf32dc6e4d7880b99f6f9d18c408]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:56.799564 28980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44bc823d03c42b690b1e3fef7371e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:56.800007 28979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87f738af6dec4151b08895d4f753fc5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:56.800393 28981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c1142c68564c98b6c942ffd8c0efdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:56.800273 29115 tablet_service.cc:1467] Processing CreateTablet for tablet 87f738af6dec4151b08895d4f753fc5e (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07faf32dc6e4d7880b99f6f9d18c408]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:56.800839 29115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87f738af6dec4151b08895d4f753fc5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:56.803158 29117 tablet_service.cc:1467] Processing CreateTablet for tablet f6c1142c68564c98b6c942ffd8c0efdc (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07faf32dc6e4d7880b99f6f9d18c408]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:56.803740 29117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c1142c68564c98b6c942ffd8c0efdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:56.807643 29251 tablet_service.cc:1467] Processing CreateTablet for tablet b44bc823d03c42b690b1e3fef7371e7b (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07faf32dc6e4d7880b99f6f9d18c408]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:56.808862 29251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44bc823d03c42b690b1e3fef7371e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:56.811424 29250 tablet_service.cc:1467] Processing CreateTablet for tablet 87f738af6dec4151b08895d4f753fc5e (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07faf32dc6e4d7880b99f6f9d18c408]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:56.811952 29250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87f738af6dec4151b08895d4f753fc5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:56.806393 29252 tablet_service.cc:1467] Processing CreateTablet for tablet f6c1142c68564c98b6c942ffd8c0efdc (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07faf32dc6e4d7880b99f6f9d18c408]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:56.823817 29252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c1142c68564c98b6c942ffd8c0efdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:56.831068 29479 tablet_bootstrap.cc:492] T b44bc823d03c42b690b1e3fef7371e7b P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Bootstrap starting.
I20240502 21:35:56.831063 29480 tablet_bootstrap.cc:492] T f6c1142c68564c98b6c942ffd8c0efdc P 8148636210184b38b1fe012cd2d0eb29: Bootstrap starting.
I20240502 21:35:56.835570 29479 tablet_bootstrap.cc:654] T b44bc823d03c42b690b1e3fef7371e7b P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:56.837097 29480 tablet_bootstrap.cc:654] T f6c1142c68564c98b6c942ffd8c0efdc P 8148636210184b38b1fe012cd2d0eb29: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:56.837266 29479 log.cc:824] T b44bc823d03c42b690b1e3fef7371e7b P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Log is configured to *not* fsync() on all Append() calls
I20240502 21:35:56.838840 29480 log.cc:824] T f6c1142c68564c98b6c942ffd8c0efdc P 8148636210184b38b1fe012cd2d0eb29: Log is configured to *not* fsync() on all Append() calls
I20240502 21:35:56.846444 29480 tablet_bootstrap.cc:492] T f6c1142c68564c98b6c942ffd8c0efdc P 8148636210184b38b1fe012cd2d0eb29: No bootstrap required, opened a new log
I20240502 21:35:56.849591 29480 ts_tablet_manager.cc:1367] T f6c1142c68564c98b6c942ffd8c0efdc P 8148636210184b38b1fe012cd2d0eb29: Time spent bootstrapping tablet: real 0.019s	user 0.008s	sys 0.005s
I20240502 21:35:56.854329 29479 tablet_bootstrap.cc:492] T b44bc823d03c42b690b1e3fef7371e7b P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: No bootstrap required, opened a new log
I20240502 21:35:56.854833 29479 ts_tablet_manager.cc:1367] T b44bc823d03c42b690b1e3fef7371e7b P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Time spent bootstrapping tablet: real 0.024s	user 0.007s	sys 0.015s
I20240502 21:35:56.860671 29483 tablet_bootstrap.cc:492] T b44bc823d03c42b690b1e3fef7371e7b P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Bootstrap starting.
I20240502 21:35:56.861294 29480 raft_consensus.cc:348] T f6c1142c68564c98b6c942ffd8c0efdc P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:56.861719 29480 raft_consensus.cc:374] T f6c1142c68564c98b6c942ffd8c0efdc P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:56.861820 29480 raft_consensus.cc:729] T f6c1142c68564c98b6c942ffd8c0efdc P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8148636210184b38b1fe012cd2d0eb29, State: Initialized, Role: FOLLOWER
I20240502 21:35:56.862514 29480 consensus_queue.cc:260] T f6c1142c68564c98b6c942ffd8c0efdc P 8148636210184b38b1fe012cd2d0eb29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:56.864506 29480 ts_tablet_manager.cc:1398] T f6c1142c68564c98b6c942ffd8c0efdc P 8148636210184b38b1fe012cd2d0eb29: Time spent starting tablet: real 0.015s	user 0.012s	sys 0.000s
I20240502 21:35:56.864704 29483 tablet_bootstrap.cc:654] T b44bc823d03c42b690b1e3fef7371e7b P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:56.865077 29480 tablet_bootstrap.cc:492] T b44bc823d03c42b690b1e3fef7371e7b P 8148636210184b38b1fe012cd2d0eb29: Bootstrap starting.
I20240502 21:35:56.866617 29480 tablet_bootstrap.cc:654] T b44bc823d03c42b690b1e3fef7371e7b P 8148636210184b38b1fe012cd2d0eb29: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:56.866863 29484 raft_consensus.cc:482] T f6c1142c68564c98b6c942ffd8c0efdc P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:35:56.867348 29483 log.cc:824] T b44bc823d03c42b690b1e3fef7371e7b P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Log is configured to *not* fsync() on all Append() calls
I20240502 21:35:56.867209 29484 raft_consensus.cc:504] T f6c1142c68564c98b6c942ffd8c0efdc P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:56.874143 29479 raft_consensus.cc:348] T b44bc823d03c42b690b1e3fef7371e7b P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:56.874855 29479 raft_consensus.cc:374] T b44bc823d03c42b690b1e3fef7371e7b P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:56.874989 29479 raft_consensus.cc:729] T b44bc823d03c42b690b1e3fef7371e7b P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da7dc5c70ff4f3fa6abf7cc75bb12f2, State: Initialized, Role: FOLLOWER
I20240502 21:35:56.876441 29479 consensus_queue.cc:260] T b44bc823d03c42b690b1e3fef7371e7b P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:56.880702 29483 tablet_bootstrap.cc:492] T b44bc823d03c42b690b1e3fef7371e7b P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: No bootstrap required, opened a new log
I20240502 21:35:56.881067 29483 ts_tablet_manager.cc:1367] T b44bc823d03c42b690b1e3fef7371e7b P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Time spent bootstrapping tablet: real 0.021s	user 0.005s	sys 0.013s
I20240502 21:35:56.884088 29484 leader_election.cc:288] T f6c1142c68564c98b6c942ffd8c0efdc P 8148636210184b38b1fe012cd2d0eb29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5da7dc5c70ff4f3fa6abf7cc75bb12f2 (127.27.196.65:42025), a3ecdc3dc21946cfb5f4a2e5ab0a75ba (127.27.196.67:36675)
I20240502 21:35:56.888715 29479 ts_tablet_manager.cc:1398] T b44bc823d03c42b690b1e3fef7371e7b P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Time spent starting tablet: real 0.034s	user 0.013s	sys 0.000s
I20240502 21:35:56.891400 29483 raft_consensus.cc:348] T b44bc823d03c42b690b1e3fef7371e7b P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:56.892005 29483 raft_consensus.cc:374] T b44bc823d03c42b690b1e3fef7371e7b P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:56.892148 29483 raft_consensus.cc:729] T b44bc823d03c42b690b1e3fef7371e7b P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3ecdc3dc21946cfb5f4a2e5ab0a75ba, State: Initialized, Role: FOLLOWER
I20240502 21:35:56.893090 29483 consensus_queue.cc:260] T b44bc823d03c42b690b1e3fef7371e7b P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:56.894122 29479 tablet_bootstrap.cc:492] T 87f738af6dec4151b08895d4f753fc5e P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Bootstrap starting.
I20240502 21:35:56.896176 29479 tablet_bootstrap.cc:654] T 87f738af6dec4151b08895d4f753fc5e P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:56.897280 29273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c1142c68564c98b6c942ffd8c0efdc" candidate_uuid: "8148636210184b38b1fe012cd2d0eb29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" is_pre_election: true
I20240502 21:35:56.898301 29480 tablet_bootstrap.cc:492] T b44bc823d03c42b690b1e3fef7371e7b P 8148636210184b38b1fe012cd2d0eb29: No bootstrap required, opened a new log
I20240502 21:35:56.898502 29480 ts_tablet_manager.cc:1367] T b44bc823d03c42b690b1e3fef7371e7b P 8148636210184b38b1fe012cd2d0eb29: Time spent bootstrapping tablet: real 0.034s	user 0.017s	sys 0.012s
W20240502 21:35:56.899082 29070 leader_election.cc:341] T f6c1142c68564c98b6c942ffd8c0efdc P 8148636210184b38b1fe012cd2d0eb29 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3ecdc3dc21946cfb5f4a2e5ab0a75ba (127.27.196.67:36675): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:35:56.899633 29480 raft_consensus.cc:348] T b44bc823d03c42b690b1e3fef7371e7b P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:56.899899 29480 raft_consensus.cc:374] T b44bc823d03c42b690b1e3fef7371e7b P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:56.900007 29480 raft_consensus.cc:729] T b44bc823d03c42b690b1e3fef7371e7b P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8148636210184b38b1fe012cd2d0eb29, State: Initialized, Role: FOLLOWER
I20240502 21:35:56.900382 29479 tablet_bootstrap.cc:492] T 87f738af6dec4151b08895d4f753fc5e P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: No bootstrap required, opened a new log
I20240502 21:35:56.900271 29480 consensus_queue.cc:260] T b44bc823d03c42b690b1e3fef7371e7b P 8148636210184b38b1fe012cd2d0eb29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:56.900619 29479 ts_tablet_manager.cc:1367] T 87f738af6dec4151b08895d4f753fc5e P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Time spent bootstrapping tablet: real 0.007s	user 0.004s	sys 0.000s
I20240502 21:35:56.900871 29480 ts_tablet_manager.cc:1398] T b44bc823d03c42b690b1e3fef7371e7b P 8148636210184b38b1fe012cd2d0eb29: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:35:56.901161 29480 tablet_bootstrap.cc:492] T 87f738af6dec4151b08895d4f753fc5e P 8148636210184b38b1fe012cd2d0eb29: Bootstrap starting.
I20240502 21:35:56.901515 29483 ts_tablet_manager.cc:1398] T b44bc823d03c42b690b1e3fef7371e7b P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Time spent starting tablet: real 0.020s	user 0.010s	sys 0.010s
I20240502 21:35:56.903008 29480 tablet_bootstrap.cc:654] T 87f738af6dec4151b08895d4f753fc5e P 8148636210184b38b1fe012cd2d0eb29: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:56.903347 29479 raft_consensus.cc:348] T 87f738af6dec4151b08895d4f753fc5e P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:56.903673 29479 raft_consensus.cc:374] T 87f738af6dec4151b08895d4f753fc5e P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:56.903828 29479 raft_consensus.cc:729] T 87f738af6dec4151b08895d4f753fc5e P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da7dc5c70ff4f3fa6abf7cc75bb12f2, State: Initialized, Role: FOLLOWER
I20240502 21:35:56.904150 29479 consensus_queue.cc:260] T 87f738af6dec4151b08895d4f753fc5e P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:56.905793 29479 ts_tablet_manager.cc:1398] T 87f738af6dec4151b08895d4f753fc5e P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 21:35:56.906213 29479 tablet_bootstrap.cc:492] T f6c1142c68564c98b6c942ffd8c0efdc P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Bootstrap starting.
I20240502 21:35:56.907925 29002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c1142c68564c98b6c942ffd8c0efdc" candidate_uuid: "8148636210184b38b1fe012cd2d0eb29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" is_pre_election: true
I20240502 21:35:56.908340 29479 tablet_bootstrap.cc:654] T f6c1142c68564c98b6c942ffd8c0efdc P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Neither blocks nor log segments found. Creating new log.
W20240502 21:35:56.909816 29071 leader_election.cc:341] T f6c1142c68564c98b6c942ffd8c0efdc P 8148636210184b38b1fe012cd2d0eb29 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5da7dc5c70ff4f3fa6abf7cc75bb12f2 (127.27.196.65:42025): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:35:56.910068 29071 leader_election.cc:302] T f6c1142c68564c98b6c942ffd8c0efdc P 8148636210184b38b1fe012cd2d0eb29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8148636210184b38b1fe012cd2d0eb29; no voters: 5da7dc5c70ff4f3fa6abf7cc75bb12f2, a3ecdc3dc21946cfb5f4a2e5ab0a75ba
I20240502 21:35:56.911330 29484 raft_consensus.cc:2726] T f6c1142c68564c98b6c942ffd8c0efdc P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:35:56.913575 29483 tablet_bootstrap.cc:492] T 87f738af6dec4151b08895d4f753fc5e P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Bootstrap starting.
I20240502 21:35:56.915992 29483 tablet_bootstrap.cc:654] T 87f738af6dec4151b08895d4f753fc5e P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:56.917009 29479 tablet_bootstrap.cc:492] T f6c1142c68564c98b6c942ffd8c0efdc P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: No bootstrap required, opened a new log
I20240502 21:35:56.917274 29479 ts_tablet_manager.cc:1367] T f6c1142c68564c98b6c942ffd8c0efdc P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Time spent bootstrapping tablet: real 0.011s	user 0.002s	sys 0.006s
I20240502 21:35:56.917479 29480 tablet_bootstrap.cc:492] T 87f738af6dec4151b08895d4f753fc5e P 8148636210184b38b1fe012cd2d0eb29: No bootstrap required, opened a new log
I20240502 21:35:56.917673 29480 ts_tablet_manager.cc:1367] T 87f738af6dec4151b08895d4f753fc5e P 8148636210184b38b1fe012cd2d0eb29: Time spent bootstrapping tablet: real 0.017s	user 0.008s	sys 0.008s
I20240502 21:35:56.918332 29480 raft_consensus.cc:348] T 87f738af6dec4151b08895d4f753fc5e P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:56.918555 29480 raft_consensus.cc:374] T 87f738af6dec4151b08895d4f753fc5e P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:56.918396 29479 raft_consensus.cc:348] T f6c1142c68564c98b6c942ffd8c0efdc P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:56.918648 29480 raft_consensus.cc:729] T 87f738af6dec4151b08895d4f753fc5e P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8148636210184b38b1fe012cd2d0eb29, State: Initialized, Role: FOLLOWER
I20240502 21:35:56.918748 29479 raft_consensus.cc:374] T f6c1142c68564c98b6c942ffd8c0efdc P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:56.918849 29479 raft_consensus.cc:729] T f6c1142c68564c98b6c942ffd8c0efdc P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da7dc5c70ff4f3fa6abf7cc75bb12f2, State: Initialized, Role: FOLLOWER
I20240502 21:35:56.918958 29480 consensus_queue.cc:260] T 87f738af6dec4151b08895d4f753fc5e P 8148636210184b38b1fe012cd2d0eb29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:56.919215 29479 consensus_queue.cc:260] T f6c1142c68564c98b6c942ffd8c0efdc P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:56.919674 29480 ts_tablet_manager.cc:1398] T 87f738af6dec4151b08895d4f753fc5e P 8148636210184b38b1fe012cd2d0eb29: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 21:35:56.919961 29479 ts_tablet_manager.cc:1398] T f6c1142c68564c98b6c942ffd8c0efdc P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 21:35:56.921475 29483 tablet_bootstrap.cc:492] T 87f738af6dec4151b08895d4f753fc5e P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: No bootstrap required, opened a new log
I20240502 21:35:56.921674 29483 ts_tablet_manager.cc:1367] T 87f738af6dec4151b08895d4f753fc5e P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Time spent bootstrapping tablet: real 0.008s	user 0.005s	sys 0.001s
I20240502 21:35:56.922746 29483 raft_consensus.cc:348] T 87f738af6dec4151b08895d4f753fc5e P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:56.922993 29483 raft_consensus.cc:374] T 87f738af6dec4151b08895d4f753fc5e P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:56.923101 29483 raft_consensus.cc:729] T 87f738af6dec4151b08895d4f753fc5e P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3ecdc3dc21946cfb5f4a2e5ab0a75ba, State: Initialized, Role: FOLLOWER
I20240502 21:35:56.923425 29483 consensus_queue.cc:260] T 87f738af6dec4151b08895d4f753fc5e P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:56.924607 29483 ts_tablet_manager.cc:1398] T 87f738af6dec4151b08895d4f753fc5e P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 21:35:56.924888 29483 tablet_bootstrap.cc:492] T f6c1142c68564c98b6c942ffd8c0efdc P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Bootstrap starting.
I20240502 21:35:56.926640 29483 tablet_bootstrap.cc:654] T f6c1142c68564c98b6c942ffd8c0efdc P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:56.928645 29483 tablet_bootstrap.cc:492] T f6c1142c68564c98b6c942ffd8c0efdc P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: No bootstrap required, opened a new log
I20240502 21:35:56.928805 29483 ts_tablet_manager.cc:1367] T f6c1142c68564c98b6c942ffd8c0efdc P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240502 21:35:56.929416 29483 raft_consensus.cc:348] T f6c1142c68564c98b6c942ffd8c0efdc P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:56.929607 29483 raft_consensus.cc:374] T f6c1142c68564c98b6c942ffd8c0efdc P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:56.929678 29483 raft_consensus.cc:729] T f6c1142c68564c98b6c942ffd8c0efdc P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3ecdc3dc21946cfb5f4a2e5ab0a75ba, State: Initialized, Role: FOLLOWER
I20240502 21:35:56.929953 29483 consensus_queue.cc:260] T f6c1142c68564c98b6c942ffd8c0efdc P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:56.930491 29483 ts_tablet_manager.cc:1398] T f6c1142c68564c98b6c942ffd8c0efdc P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
W20240502 21:35:56.931522 29186 tablet.cc:2369] T 87f738af6dec4151b08895d4f753fc5e P 8148636210184b38b1fe012cd2d0eb29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:35:56.969818 29484 raft_consensus.cc:482] T b44bc823d03c42b690b1e3fef7371e7b P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:35:56.970042 29484 raft_consensus.cc:504] T b44bc823d03c42b690b1e3fef7371e7b P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:56.970846 29484 leader_election.cc:288] T b44bc823d03c42b690b1e3fef7371e7b P 8148636210184b38b1fe012cd2d0eb29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5da7dc5c70ff4f3fa6abf7cc75bb12f2 (127.27.196.65:42025), a3ecdc3dc21946cfb5f4a2e5ab0a75ba (127.27.196.67:36675)
I20240502 21:35:56.971496 29002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44bc823d03c42b690b1e3fef7371e7b" candidate_uuid: "8148636210184b38b1fe012cd2d0eb29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" is_pre_election: true
I20240502 21:35:56.971544 29273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44bc823d03c42b690b1e3fef7371e7b" candidate_uuid: "8148636210184b38b1fe012cd2d0eb29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" is_pre_election: true
I20240502 21:35:56.971936 29273 raft_consensus.cc:2454] T b44bc823d03c42b690b1e3fef7371e7b P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8148636210184b38b1fe012cd2d0eb29 in term 0.
I20240502 21:35:56.972086 29002 raft_consensus.cc:2454] T b44bc823d03c42b690b1e3fef7371e7b P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8148636210184b38b1fe012cd2d0eb29 in term 0.
I20240502 21:35:56.972608 29070 leader_election.cc:302] T b44bc823d03c42b690b1e3fef7371e7b P 8148636210184b38b1fe012cd2d0eb29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8148636210184b38b1fe012cd2d0eb29, a3ecdc3dc21946cfb5f4a2e5ab0a75ba; no voters: 
I20240502 21:35:56.973031 29484 raft_consensus.cc:2781] T b44bc823d03c42b690b1e3fef7371e7b P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:35:56.973163 29484 raft_consensus.cc:482] T b44bc823d03c42b690b1e3fef7371e7b P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:35:56.973260 29484 raft_consensus.cc:3037] T b44bc823d03c42b690b1e3fef7371e7b P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:56.976033 29484 raft_consensus.cc:504] T b44bc823d03c42b690b1e3fef7371e7b P 8148636210184b38b1fe012cd2d0eb29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:56.976289 29490 raft_consensus.cc:482] T 87f738af6dec4151b08895d4f753fc5e P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 21:35:56.976467 29322 tablet.cc:2369] T 87f738af6dec4151b08895d4f753fc5e P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:35:56.976506 29490 raft_consensus.cc:504] T 87f738af6dec4151b08895d4f753fc5e P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:56.976995 29484 leader_election.cc:288] T b44bc823d03c42b690b1e3fef7371e7b P 8148636210184b38b1fe012cd2d0eb29 [CANDIDATE]: Term 1 election: Requested vote from peers 5da7dc5c70ff4f3fa6abf7cc75bb12f2 (127.27.196.65:42025), a3ecdc3dc21946cfb5f4a2e5ab0a75ba (127.27.196.67:36675)
I20240502 21:35:56.978744 29490 leader_election.cc:288] T 87f738af6dec4151b08895d4f753fc5e P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5da7dc5c70ff4f3fa6abf7cc75bb12f2 (127.27.196.65:42025), 8148636210184b38b1fe012cd2d0eb29 (127.27.196.66:38577)
I20240502 21:35:56.978777 29002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44bc823d03c42b690b1e3fef7371e7b" candidate_uuid: "8148636210184b38b1fe012cd2d0eb29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2"
I20240502 21:35:56.979058 29002 raft_consensus.cc:3037] T b44bc823d03c42b690b1e3fef7371e7b P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:56.982054 29273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44bc823d03c42b690b1e3fef7371e7b" candidate_uuid: "8148636210184b38b1fe012cd2d0eb29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba"
I20240502 21:35:56.982328 29273 raft_consensus.cc:3037] T b44bc823d03c42b690b1e3fef7371e7b P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:56.983011 29002 raft_consensus.cc:2454] T b44bc823d03c42b690b1e3fef7371e7b P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8148636210184b38b1fe012cd2d0eb29 in term 1.
I20240502 21:35:56.984375 29071 leader_election.cc:302] T b44bc823d03c42b690b1e3fef7371e7b P 8148636210184b38b1fe012cd2d0eb29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5da7dc5c70ff4f3fa6abf7cc75bb12f2, 8148636210184b38b1fe012cd2d0eb29; no voters: 
I20240502 21:35:56.984853 29484 raft_consensus.cc:2781] T b44bc823d03c42b690b1e3fef7371e7b P 8148636210184b38b1fe012cd2d0eb29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:56.985704 29273 raft_consensus.cc:2454] T b44bc823d03c42b690b1e3fef7371e7b P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8148636210184b38b1fe012cd2d0eb29 in term 1.
I20240502 21:35:56.986963 29484 raft_consensus.cc:686] T b44bc823d03c42b690b1e3fef7371e7b P 8148636210184b38b1fe012cd2d0eb29 [term 1 LEADER]: Becoming Leader. State: Replica: 8148636210184b38b1fe012cd2d0eb29, State: Running, Role: LEADER
I20240502 21:35:56.987790 29484 consensus_queue.cc:237] T b44bc823d03c42b690b1e3fef7371e7b P 8148636210184b38b1fe012cd2d0eb29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:56.994746 29002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f738af6dec4151b08895d4f753fc5e" candidate_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" is_pre_election: true
I20240502 21:35:56.995076 29002 raft_consensus.cc:2454] T 87f738af6dec4151b08895d4f753fc5e P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3ecdc3dc21946cfb5f4a2e5ab0a75ba in term 0.
I20240502 21:35:56.994626 28854 catalog_manager.cc:5484] T b44bc823d03c42b690b1e3fef7371e7b P 8148636210184b38b1fe012cd2d0eb29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8148636210184b38b1fe012cd2d0eb29 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "8148636210184b38b1fe012cd2d0eb29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } health_report { overall_health: UNKNOWN } } }
I20240502 21:35:56.997299 29137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f738af6dec4151b08895d4f753fc5e" candidate_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8148636210184b38b1fe012cd2d0eb29" is_pre_election: true
I20240502 21:35:56.997707 29137 raft_consensus.cc:2454] T 87f738af6dec4151b08895d4f753fc5e P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3ecdc3dc21946cfb5f4a2e5ab0a75ba in term 0.
I20240502 21:35:56.997798 29207 leader_election.cc:302] T 87f738af6dec4151b08895d4f753fc5e P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5da7dc5c70ff4f3fa6abf7cc75bb12f2, a3ecdc3dc21946cfb5f4a2e5ab0a75ba; no voters: 
I20240502 21:35:56.998317 29490 raft_consensus.cc:2781] T 87f738af6dec4151b08895d4f753fc5e P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:35:56.998473 29490 raft_consensus.cc:482] T 87f738af6dec4151b08895d4f753fc5e P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:35:56.998572 29490 raft_consensus.cc:3037] T 87f738af6dec4151b08895d4f753fc5e P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:57.001014 29490 raft_consensus.cc:504] T 87f738af6dec4151b08895d4f753fc5e P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:57.001716 29490 leader_election.cc:288] T 87f738af6dec4151b08895d4f753fc5e P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [CANDIDATE]: Term 1 election: Requested vote from peers 5da7dc5c70ff4f3fa6abf7cc75bb12f2 (127.27.196.65:42025), 8148636210184b38b1fe012cd2d0eb29 (127.27.196.66:38577)
I20240502 21:35:57.002331 29002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f738af6dec4151b08895d4f753fc5e" candidate_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2"
I20240502 21:35:57.002394 29137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f738af6dec4151b08895d4f753fc5e" candidate_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8148636210184b38b1fe012cd2d0eb29"
I20240502 21:35:57.002537 29002 raft_consensus.cc:3037] T 87f738af6dec4151b08895d4f753fc5e P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:57.002632 29137 raft_consensus.cc:3037] T 87f738af6dec4151b08895d4f753fc5e P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:57.004793 29137 raft_consensus.cc:2454] T 87f738af6dec4151b08895d4f753fc5e P 8148636210184b38b1fe012cd2d0eb29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3ecdc3dc21946cfb5f4a2e5ab0a75ba in term 1.
I20240502 21:35:57.004971 29002 raft_consensus.cc:2454] T 87f738af6dec4151b08895d4f753fc5e P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3ecdc3dc21946cfb5f4a2e5ab0a75ba in term 1.
I20240502 21:35:57.005465 29206 leader_election.cc:302] T 87f738af6dec4151b08895d4f753fc5e P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8148636210184b38b1fe012cd2d0eb29, a3ecdc3dc21946cfb5f4a2e5ab0a75ba; no voters: 
I20240502 21:35:57.005831 29490 raft_consensus.cc:2781] T 87f738af6dec4151b08895d4f753fc5e P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:57.006448 29490 raft_consensus.cc:686] T 87f738af6dec4151b08895d4f753fc5e P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 1 LEADER]: Becoming Leader. State: Replica: a3ecdc3dc21946cfb5f4a2e5ab0a75ba, State: Running, Role: LEADER
I20240502 21:35:57.006985 29490 consensus_queue.cc:237] T 87f738af6dec4151b08895d4f753fc5e P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:57.012717 28854 catalog_manager.cc:5484] T 87f738af6dec4151b08895d4f753fc5e P a3ecdc3dc21946cfb5f4a2e5ab0a75ba reported cstate change: term changed from 0 to 1, leader changed from <none> to a3ecdc3dc21946cfb5f4a2e5ab0a75ba (127.27.196.67). New cstate: current_term: 1 leader_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } health_report { overall_health: HEALTHY } } }
I20240502 21:35:57.026069 29488 raft_consensus.cc:482] T f6c1142c68564c98b6c942ffd8c0efdc P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:35:57.026355 29488 raft_consensus.cc:504] T f6c1142c68564c98b6c942ffd8c0efdc P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:57.028357 29488 leader_election.cc:288] T f6c1142c68564c98b6c942ffd8c0efdc P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8148636210184b38b1fe012cd2d0eb29 (127.27.196.66:38577), a3ecdc3dc21946cfb5f4a2e5ab0a75ba (127.27.196.67:36675)
I20240502 21:35:57.037400 29137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c1142c68564c98b6c942ffd8c0efdc" candidate_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8148636210184b38b1fe012cd2d0eb29" is_pre_election: true
I20240502 21:35:57.037757 29137 raft_consensus.cc:2454] T f6c1142c68564c98b6c942ffd8c0efdc P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5da7dc5c70ff4f3fa6abf7cc75bb12f2 in term 0.
I20240502 21:35:57.038118 29273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c1142c68564c98b6c942ffd8c0efdc" candidate_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" is_pre_election: true
I20240502 21:35:57.038415 29273 raft_consensus.cc:2454] T f6c1142c68564c98b6c942ffd8c0efdc P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5da7dc5c70ff4f3fa6abf7cc75bb12f2 in term 0.
I20240502 21:35:57.038408 28933 leader_election.cc:302] T f6c1142c68564c98b6c942ffd8c0efdc P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5da7dc5c70ff4f3fa6abf7cc75bb12f2, 8148636210184b38b1fe012cd2d0eb29; no voters: 
I20240502 21:35:57.038911 29488 raft_consensus.cc:2781] T f6c1142c68564c98b6c942ffd8c0efdc P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:35:57.039063 29488 raft_consensus.cc:482] T f6c1142c68564c98b6c942ffd8c0efdc P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:35:57.039199 29488 raft_consensus.cc:3037] T f6c1142c68564c98b6c942ffd8c0efdc P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:57.041378 29488 raft_consensus.cc:504] T f6c1142c68564c98b6c942ffd8c0efdc P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:57.042039 29488 leader_election.cc:288] T f6c1142c68564c98b6c942ffd8c0efdc P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [CANDIDATE]: Term 1 election: Requested vote from peers 8148636210184b38b1fe012cd2d0eb29 (127.27.196.66:38577), a3ecdc3dc21946cfb5f4a2e5ab0a75ba (127.27.196.67:36675)
I20240502 21:35:57.042668 29137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c1142c68564c98b6c942ffd8c0efdc" candidate_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8148636210184b38b1fe012cd2d0eb29"
I20240502 21:35:57.042721 29273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c1142c68564c98b6c942ffd8c0efdc" candidate_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba"
I20240502 21:35:57.042932 29273 raft_consensus.cc:3037] T f6c1142c68564c98b6c942ffd8c0efdc P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:57.042932 29137 raft_consensus.cc:3037] T f6c1142c68564c98b6c942ffd8c0efdc P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:57.045998 29273 raft_consensus.cc:2454] T f6c1142c68564c98b6c942ffd8c0efdc P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5da7dc5c70ff4f3fa6abf7cc75bb12f2 in term 1.
I20240502 21:35:57.045993 29137 raft_consensus.cc:2454] T f6c1142c68564c98b6c942ffd8c0efdc P 8148636210184b38b1fe012cd2d0eb29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5da7dc5c70ff4f3fa6abf7cc75bb12f2 in term 1.
I20240502 21:35:57.046564 28933 leader_election.cc:302] T f6c1142c68564c98b6c942ffd8c0efdc P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5da7dc5c70ff4f3fa6abf7cc75bb12f2, a3ecdc3dc21946cfb5f4a2e5ab0a75ba; no voters: 
I20240502 21:35:57.046912 29488 raft_consensus.cc:2781] T f6c1142c68564c98b6c942ffd8c0efdc P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:57.047537 29488 raft_consensus.cc:686] T f6c1142c68564c98b6c942ffd8c0efdc P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 1 LEADER]: Becoming Leader. State: Replica: 5da7dc5c70ff4f3fa6abf7cc75bb12f2, State: Running, Role: LEADER
I20240502 21:35:57.048185 29488 consensus_queue.cc:237] T f6c1142c68564c98b6c942ffd8c0efdc P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:57.053282 28854 catalog_manager.cc:5484] T f6c1142c68564c98b6c942ffd8c0efdc P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5da7dc5c70ff4f3fa6abf7cc75bb12f2 (127.27.196.65). New cstate: current_term: 1 leader_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } health_report { overall_health: UNKNOWN } } }
W20240502 21:35:57.079332 29050 tablet.cc:2369] T 87f738af6dec4151b08895d4f753fc5e P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:35:57.445706 29484 consensus_queue.cc:1035] T b44bc823d03c42b690b1e3fef7371e7b P 8148636210184b38b1fe012cd2d0eb29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:35:57.455194 29484 consensus_queue.cc:1035] T b44bc823d03c42b690b1e3fef7371e7b P 8148636210184b38b1fe012cd2d0eb29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:35:57.468029 29488 consensus_queue.cc:1035] T f6c1142c68564c98b6c942ffd8c0efdc P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:35:57.474073 29504 consensus_queue.cc:1035] T f6c1142c68564c98b6c942ffd8c0efdc P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:35:57.504925 29506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:35:57.505327 29506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:35:57.513597 29506 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:35:57.556957 29490 consensus_queue.cc:1035] T 87f738af6dec4151b08895d4f753fc5e P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240502 21:35:57.565045 29530 consensus_queue.cc:1035] T 87f738af6dec4151b08895d4f753fc5e P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 321.465 ms
  time per row: 0.160733 ms
I20240502 21:35:58.018168 28853 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59708:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240502 21:35:58.019040 28853 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 21:35:58.030292 29117 tablet_service.cc:1467] Processing CreateTablet for tablet ef659f7850914bb289fac74b6741544c (DEFAULT_TABLE table=rebalance_test_table_1 [id=75bc205971214a6a8e552a8528d72b8f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:58.030588 28979 tablet_service.cc:1467] Processing CreateTablet for tablet ef659f7850914bb289fac74b6741544c (DEFAULT_TABLE table=rebalance_test_table_1 [id=75bc205971214a6a8e552a8528d72b8f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:58.030891 29117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef659f7850914bb289fac74b6741544c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:58.030820 29251 tablet_service.cc:1467] Processing CreateTablet for tablet ef659f7850914bb289fac74b6741544c (DEFAULT_TABLE table=rebalance_test_table_1 [id=75bc205971214a6a8e552a8528d72b8f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:58.031386 28979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef659f7850914bb289fac74b6741544c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:58.031455 29251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef659f7850914bb289fac74b6741544c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:58.034971 28980 tablet_service.cc:1467] Processing CreateTablet for tablet 8db8c5faefbf454fa68d626a8e523654 (DEFAULT_TABLE table=rebalance_test_table_1 [id=75bc205971214a6a8e552a8528d72b8f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:58.035447 28980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db8c5faefbf454fa68d626a8e523654. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:58.036171 28981 tablet_service.cc:1467] Processing CreateTablet for tablet bc25546ca0fe4e44a6edf394a6e13aa1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=75bc205971214a6a8e552a8528d72b8f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:58.036568 28981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc25546ca0fe4e44a6edf394a6e13aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:58.039649 29115 tablet_service.cc:1467] Processing CreateTablet for tablet bc25546ca0fe4e44a6edf394a6e13aa1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=75bc205971214a6a8e552a8528d72b8f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:58.040165 29115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc25546ca0fe4e44a6edf394a6e13aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:58.043720 29116 tablet_service.cc:1467] Processing CreateTablet for tablet 8db8c5faefbf454fa68d626a8e523654 (DEFAULT_TABLE table=rebalance_test_table_1 [id=75bc205971214a6a8e552a8528d72b8f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:58.044296 29116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db8c5faefbf454fa68d626a8e523654. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:58.047819 29250 tablet_service.cc:1467] Processing CreateTablet for tablet bc25546ca0fe4e44a6edf394a6e13aa1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=75bc205971214a6a8e552a8528d72b8f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:58.048425 29250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc25546ca0fe4e44a6edf394a6e13aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:58.048902 29575 tablet_bootstrap.cc:492] T ef659f7850914bb289fac74b6741544c P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Bootstrap starting.
I20240502 21:35:58.051842 29252 tablet_service.cc:1467] Processing CreateTablet for tablet 8db8c5faefbf454fa68d626a8e523654 (DEFAULT_TABLE table=rebalance_test_table_1 [id=75bc205971214a6a8e552a8528d72b8f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:58.052481 29575 tablet_bootstrap.cc:654] T ef659f7850914bb289fac74b6741544c P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:58.052487 29252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db8c5faefbf454fa68d626a8e523654. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:58.056721 29576 tablet_bootstrap.cc:492] T 8db8c5faefbf454fa68d626a8e523654 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Bootstrap starting.
I20240502 21:35:58.060266 29576 tablet_bootstrap.cc:654] T 8db8c5faefbf454fa68d626a8e523654 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:58.061897 29575 tablet_bootstrap.cc:492] T ef659f7850914bb289fac74b6741544c P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: No bootstrap required, opened a new log
I20240502 21:35:58.062103 29575 ts_tablet_manager.cc:1367] T ef659f7850914bb289fac74b6741544c P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Time spent bootstrapping tablet: real 0.013s	user 0.006s	sys 0.000s
I20240502 21:35:58.065115 29575 raft_consensus.cc:348] T ef659f7850914bb289fac74b6741544c P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:58.065768 29575 raft_consensus.cc:374] T ef659f7850914bb289fac74b6741544c P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:58.065930 29575 raft_consensus.cc:729] T ef659f7850914bb289fac74b6741544c P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3ecdc3dc21946cfb5f4a2e5ab0a75ba, State: Initialized, Role: FOLLOWER
I20240502 21:35:58.066627 29576 tablet_bootstrap.cc:492] T 8db8c5faefbf454fa68d626a8e523654 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: No bootstrap required, opened a new log
I20240502 21:35:58.066952 29576 ts_tablet_manager.cc:1367] T 8db8c5faefbf454fa68d626a8e523654 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Time spent bootstrapping tablet: real 0.010s	user 0.004s	sys 0.003s
I20240502 21:35:58.068198 29575 consensus_queue.cc:260] T ef659f7850914bb289fac74b6741544c P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:58.068449 29576 raft_consensus.cc:348] T 8db8c5faefbf454fa68d626a8e523654 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:58.069633 29575 ts_tablet_manager.cc:1398] T ef659f7850914bb289fac74b6741544c P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.000s
I20240502 21:35:58.070199 29576 raft_consensus.cc:374] T 8db8c5faefbf454fa68d626a8e523654 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:58.070452 29576 raft_consensus.cc:729] T 8db8c5faefbf454fa68d626a8e523654 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da7dc5c70ff4f3fa6abf7cc75bb12f2, State: Initialized, Role: FOLLOWER
I20240502 21:35:58.070909 29576 consensus_queue.cc:260] T 8db8c5faefbf454fa68d626a8e523654 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:58.072126 29576 ts_tablet_manager.cc:1398] T 8db8c5faefbf454fa68d626a8e523654 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.001s
I20240502 21:35:58.078550 29575 tablet_bootstrap.cc:492] T bc25546ca0fe4e44a6edf394a6e13aa1 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Bootstrap starting.
I20240502 21:35:58.080780 29576 tablet_bootstrap.cc:492] T ef659f7850914bb289fac74b6741544c P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Bootstrap starting.
I20240502 21:35:58.084852 29576 tablet_bootstrap.cc:654] T ef659f7850914bb289fac74b6741544c P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:58.086335 29574 tablet_bootstrap.cc:492] T ef659f7850914bb289fac74b6741544c P 8148636210184b38b1fe012cd2d0eb29: Bootstrap starting.
I20240502 21:35:58.087603 29575 tablet_bootstrap.cc:654] T bc25546ca0fe4e44a6edf394a6e13aa1 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:58.088052 29576 tablet_bootstrap.cc:492] T ef659f7850914bb289fac74b6741544c P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: No bootstrap required, opened a new log
I20240502 21:35:58.088268 29576 ts_tablet_manager.cc:1367] T ef659f7850914bb289fac74b6741544c P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Time spent bootstrapping tablet: real 0.008s	user 0.004s	sys 0.001s
I20240502 21:35:58.088410 29574 tablet_bootstrap.cc:654] T ef659f7850914bb289fac74b6741544c P 8148636210184b38b1fe012cd2d0eb29: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:58.089376 29576 raft_consensus.cc:348] T ef659f7850914bb289fac74b6741544c P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:58.089735 29576 raft_consensus.cc:374] T ef659f7850914bb289fac74b6741544c P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:58.089828 29576 raft_consensus.cc:729] T ef659f7850914bb289fac74b6741544c P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da7dc5c70ff4f3fa6abf7cc75bb12f2, State: Initialized, Role: FOLLOWER
I20240502 21:35:58.090176 29576 consensus_queue.cc:260] T ef659f7850914bb289fac74b6741544c P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:58.092099 29576 ts_tablet_manager.cc:1398] T ef659f7850914bb289fac74b6741544c P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.001s
I20240502 21:35:58.092561 29576 tablet_bootstrap.cc:492] T bc25546ca0fe4e44a6edf394a6e13aa1 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Bootstrap starting.
I20240502 21:35:58.096038 29576 tablet_bootstrap.cc:654] T bc25546ca0fe4e44a6edf394a6e13aa1 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:58.096738 29575 tablet_bootstrap.cc:492] T bc25546ca0fe4e44a6edf394a6e13aa1 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: No bootstrap required, opened a new log
I20240502 21:35:58.097035 29575 ts_tablet_manager.cc:1367] T bc25546ca0fe4e44a6edf394a6e13aa1 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Time spent bootstrapping tablet: real 0.019s	user 0.011s	sys 0.000s
I20240502 21:35:58.098261 29575 raft_consensus.cc:348] T bc25546ca0fe4e44a6edf394a6e13aa1 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:58.098757 29575 raft_consensus.cc:374] T bc25546ca0fe4e44a6edf394a6e13aa1 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:58.098908 29575 raft_consensus.cc:729] T bc25546ca0fe4e44a6edf394a6e13aa1 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3ecdc3dc21946cfb5f4a2e5ab0a75ba, State: Initialized, Role: FOLLOWER
I20240502 21:35:58.099438 29575 consensus_queue.cc:260] T bc25546ca0fe4e44a6edf394a6e13aa1 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:58.101471 29575 ts_tablet_manager.cc:1398] T bc25546ca0fe4e44a6edf394a6e13aa1 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 21:35:58.102075 29575 tablet_bootstrap.cc:492] T 8db8c5faefbf454fa68d626a8e523654 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Bootstrap starting.
I20240502 21:35:58.105894 29575 tablet_bootstrap.cc:654] T 8db8c5faefbf454fa68d626a8e523654 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:58.110760 29576 tablet_bootstrap.cc:492] T bc25546ca0fe4e44a6edf394a6e13aa1 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: No bootstrap required, opened a new log
I20240502 21:35:58.110755 29574 tablet_bootstrap.cc:492] T ef659f7850914bb289fac74b6741544c P 8148636210184b38b1fe012cd2d0eb29: No bootstrap required, opened a new log
I20240502 21:35:58.110937 29574 ts_tablet_manager.cc:1367] T ef659f7850914bb289fac74b6741544c P 8148636210184b38b1fe012cd2d0eb29: Time spent bootstrapping tablet: real 0.025s	user 0.012s	sys 0.012s
I20240502 21:35:58.110966 29576 ts_tablet_manager.cc:1367] T bc25546ca0fe4e44a6edf394a6e13aa1 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Time spent bootstrapping tablet: real 0.019s	user 0.008s	sys 0.008s
I20240502 21:35:58.111949 29576 raft_consensus.cc:348] T bc25546ca0fe4e44a6edf394a6e13aa1 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:58.111937 29574 raft_consensus.cc:348] T ef659f7850914bb289fac74b6741544c P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:58.112155 29574 raft_consensus.cc:374] T ef659f7850914bb289fac74b6741544c P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:58.112149 29576 raft_consensus.cc:374] T bc25546ca0fe4e44a6edf394a6e13aa1 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:58.112238 29574 raft_consensus.cc:729] T ef659f7850914bb289fac74b6741544c P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8148636210184b38b1fe012cd2d0eb29, State: Initialized, Role: FOLLOWER
I20240502 21:35:58.112253 29576 raft_consensus.cc:729] T bc25546ca0fe4e44a6edf394a6e13aa1 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da7dc5c70ff4f3fa6abf7cc75bb12f2, State: Initialized, Role: FOLLOWER
I20240502 21:35:58.112526 29576 consensus_queue.cc:260] T bc25546ca0fe4e44a6edf394a6e13aa1 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:58.112476 29574 consensus_queue.cc:260] T ef659f7850914bb289fac74b6741544c P 8148636210184b38b1fe012cd2d0eb29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:58.113497 29576 ts_tablet_manager.cc:1398] T bc25546ca0fe4e44a6edf394a6e13aa1 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 21:35:58.113593 29574 ts_tablet_manager.cc:1398] T ef659f7850914bb289fac74b6741544c P 8148636210184b38b1fe012cd2d0eb29: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.003s
I20240502 21:35:58.113934 29574 tablet_bootstrap.cc:492] T bc25546ca0fe4e44a6edf394a6e13aa1 P 8148636210184b38b1fe012cd2d0eb29: Bootstrap starting.
I20240502 21:35:58.115080 29575 tablet_bootstrap.cc:492] T 8db8c5faefbf454fa68d626a8e523654 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: No bootstrap required, opened a new log
I20240502 21:35:58.115475 29575 ts_tablet_manager.cc:1367] T 8db8c5faefbf454fa68d626a8e523654 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Time spent bootstrapping tablet: real 0.014s	user 0.002s	sys 0.004s
I20240502 21:35:58.115885 29574 tablet_bootstrap.cc:654] T bc25546ca0fe4e44a6edf394a6e13aa1 P 8148636210184b38b1fe012cd2d0eb29: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:58.116565 29575 raft_consensus.cc:348] T 8db8c5faefbf454fa68d626a8e523654 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:58.116888 29575 raft_consensus.cc:374] T 8db8c5faefbf454fa68d626a8e523654 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:58.117062 29575 raft_consensus.cc:729] T 8db8c5faefbf454fa68d626a8e523654 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3ecdc3dc21946cfb5f4a2e5ab0a75ba, State: Initialized, Role: FOLLOWER
I20240502 21:35:58.117406 29575 consensus_queue.cc:260] T 8db8c5faefbf454fa68d626a8e523654 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:58.118846 29575 ts_tablet_manager.cc:1398] T 8db8c5faefbf454fa68d626a8e523654 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 21:35:58.124496 29574 tablet_bootstrap.cc:492] T bc25546ca0fe4e44a6edf394a6e13aa1 P 8148636210184b38b1fe012cd2d0eb29: No bootstrap required, opened a new log
I20240502 21:35:58.124748 29574 ts_tablet_manager.cc:1367] T bc25546ca0fe4e44a6edf394a6e13aa1 P 8148636210184b38b1fe012cd2d0eb29: Time spent bootstrapping tablet: real 0.011s	user 0.008s	sys 0.001s
I20240502 21:35:58.125823 29574 raft_consensus.cc:348] T bc25546ca0fe4e44a6edf394a6e13aa1 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:58.126114 29574 raft_consensus.cc:374] T bc25546ca0fe4e44a6edf394a6e13aa1 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:58.126219 29574 raft_consensus.cc:729] T bc25546ca0fe4e44a6edf394a6e13aa1 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8148636210184b38b1fe012cd2d0eb29, State: Initialized, Role: FOLLOWER
I20240502 21:35:58.126549 29574 consensus_queue.cc:260] T bc25546ca0fe4e44a6edf394a6e13aa1 P 8148636210184b38b1fe012cd2d0eb29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:58.127085 29574 ts_tablet_manager.cc:1398] T bc25546ca0fe4e44a6edf394a6e13aa1 P 8148636210184b38b1fe012cd2d0eb29: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 21:35:58.127486 29574 tablet_bootstrap.cc:492] T 8db8c5faefbf454fa68d626a8e523654 P 8148636210184b38b1fe012cd2d0eb29: Bootstrap starting.
I20240502 21:35:58.129190 29574 tablet_bootstrap.cc:654] T 8db8c5faefbf454fa68d626a8e523654 P 8148636210184b38b1fe012cd2d0eb29: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:58.131610 29574 tablet_bootstrap.cc:492] T 8db8c5faefbf454fa68d626a8e523654 P 8148636210184b38b1fe012cd2d0eb29: No bootstrap required, opened a new log
I20240502 21:35:58.131779 29574 ts_tablet_manager.cc:1367] T 8db8c5faefbf454fa68d626a8e523654 P 8148636210184b38b1fe012cd2d0eb29: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.002s
I20240502 21:35:58.132534 29574 raft_consensus.cc:348] T 8db8c5faefbf454fa68d626a8e523654 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:58.132716 29574 raft_consensus.cc:374] T 8db8c5faefbf454fa68d626a8e523654 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:58.132820 29574 raft_consensus.cc:729] T 8db8c5faefbf454fa68d626a8e523654 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8148636210184b38b1fe012cd2d0eb29, State: Initialized, Role: FOLLOWER
I20240502 21:35:58.133100 29574 consensus_queue.cc:260] T 8db8c5faefbf454fa68d626a8e523654 P 8148636210184b38b1fe012cd2d0eb29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:58.133800 29574 ts_tablet_manager.cc:1398] T 8db8c5faefbf454fa68d626a8e523654 P 8148636210184b38b1fe012cd2d0eb29: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:35:58.136646 29504 raft_consensus.cc:482] T ef659f7850914bb289fac74b6741544c P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:35:58.136860 29504 raft_consensus.cc:504] T ef659f7850914bb289fac74b6741544c P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:58.137639 29504 leader_election.cc:288] T ef659f7850914bb289fac74b6741544c P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8148636210184b38b1fe012cd2d0eb29 (127.27.196.66:38577), a3ecdc3dc21946cfb5f4a2e5ab0a75ba (127.27.196.67:36675)
I20240502 21:35:58.138159 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef659f7850914bb289fac74b6741544c" candidate_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8148636210184b38b1fe012cd2d0eb29" is_pre_election: true
I20240502 21:35:58.138393 29136 raft_consensus.cc:2454] T ef659f7850914bb289fac74b6741544c P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5da7dc5c70ff4f3fa6abf7cc75bb12f2 in term 0.
I20240502 21:35:58.138391 29273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef659f7850914bb289fac74b6741544c" candidate_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" is_pre_election: true
I20240502 21:35:58.138691 29273 raft_consensus.cc:2454] T ef659f7850914bb289fac74b6741544c P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5da7dc5c70ff4f3fa6abf7cc75bb12f2 in term 0.
I20240502 21:35:58.138890 28933 leader_election.cc:302] T ef659f7850914bb289fac74b6741544c P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5da7dc5c70ff4f3fa6abf7cc75bb12f2, 8148636210184b38b1fe012cd2d0eb29; no voters: 
I20240502 21:35:58.139282 29504 raft_consensus.cc:2781] T ef659f7850914bb289fac74b6741544c P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:35:58.139385 29504 raft_consensus.cc:482] T ef659f7850914bb289fac74b6741544c P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:35:58.139523 29504 raft_consensus.cc:3037] T ef659f7850914bb289fac74b6741544c P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:58.141505 29504 raft_consensus.cc:504] T ef659f7850914bb289fac74b6741544c P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:58.142200 29504 leader_election.cc:288] T ef659f7850914bb289fac74b6741544c P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [CANDIDATE]: Term 1 election: Requested vote from peers 8148636210184b38b1fe012cd2d0eb29 (127.27.196.66:38577), a3ecdc3dc21946cfb5f4a2e5ab0a75ba (127.27.196.67:36675)
I20240502 21:35:58.142820 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef659f7850914bb289fac74b6741544c" candidate_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8148636210184b38b1fe012cd2d0eb29"
I20240502 21:35:58.142843 29273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef659f7850914bb289fac74b6741544c" candidate_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba"
I20240502 21:35:58.143038 29136 raft_consensus.cc:3037] T ef659f7850914bb289fac74b6741544c P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:58.143060 29273 raft_consensus.cc:3037] T ef659f7850914bb289fac74b6741544c P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:58.145133 29273 raft_consensus.cc:2454] T ef659f7850914bb289fac74b6741544c P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5da7dc5c70ff4f3fa6abf7cc75bb12f2 in term 1.
I20240502 21:35:58.145148 29136 raft_consensus.cc:2454] T ef659f7850914bb289fac74b6741544c P 8148636210184b38b1fe012cd2d0eb29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5da7dc5c70ff4f3fa6abf7cc75bb12f2 in term 1.
I20240502 21:35:58.145694 28933 leader_election.cc:302] T ef659f7850914bb289fac74b6741544c P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5da7dc5c70ff4f3fa6abf7cc75bb12f2, a3ecdc3dc21946cfb5f4a2e5ab0a75ba; no voters: 
I20240502 21:35:58.146107 29504 raft_consensus.cc:2781] T ef659f7850914bb289fac74b6741544c P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:58.146297 29504 raft_consensus.cc:686] T ef659f7850914bb289fac74b6741544c P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 1 LEADER]: Becoming Leader. State: Replica: 5da7dc5c70ff4f3fa6abf7cc75bb12f2, State: Running, Role: LEADER
I20240502 21:35:58.146857 29504 consensus_queue.cc:237] T ef659f7850914bb289fac74b6741544c P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:58.150830 28854 catalog_manager.cc:5484] T ef659f7850914bb289fac74b6741544c P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5da7dc5c70ff4f3fa6abf7cc75bb12f2 (127.27.196.65). New cstate: current_term: 1 leader_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } health_report { overall_health: UNKNOWN } } }
I20240502 21:35:58.152746 29484 raft_consensus.cc:482] T 8db8c5faefbf454fa68d626a8e523654 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:35:58.153021 29484 raft_consensus.cc:504] T 8db8c5faefbf454fa68d626a8e523654 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:58.153997 29484 leader_election.cc:288] T 8db8c5faefbf454fa68d626a8e523654 P 8148636210184b38b1fe012cd2d0eb29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5da7dc5c70ff4f3fa6abf7cc75bb12f2 (127.27.196.65:42025), a3ecdc3dc21946cfb5f4a2e5ab0a75ba (127.27.196.67:36675)
I20240502 21:35:58.154937 29002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db8c5faefbf454fa68d626a8e523654" candidate_uuid: "8148636210184b38b1fe012cd2d0eb29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" is_pre_election: true
I20240502 21:35:58.155443 29002 raft_consensus.cc:2454] T 8db8c5faefbf454fa68d626a8e523654 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8148636210184b38b1fe012cd2d0eb29 in term 0.
I20240502 21:35:58.156131 29273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db8c5faefbf454fa68d626a8e523654" candidate_uuid: "8148636210184b38b1fe012cd2d0eb29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" is_pre_election: true
I20240502 21:35:58.156453 29273 raft_consensus.cc:2454] T 8db8c5faefbf454fa68d626a8e523654 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8148636210184b38b1fe012cd2d0eb29 in term 0.
I20240502 21:35:58.156445 29071 leader_election.cc:302] T 8db8c5faefbf454fa68d626a8e523654 P 8148636210184b38b1fe012cd2d0eb29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5da7dc5c70ff4f3fa6abf7cc75bb12f2, 8148636210184b38b1fe012cd2d0eb29; no voters: 
I20240502 21:35:58.156880 29484 raft_consensus.cc:2781] T 8db8c5faefbf454fa68d626a8e523654 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:35:58.157002 29484 raft_consensus.cc:482] T 8db8c5faefbf454fa68d626a8e523654 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:35:58.157121 29484 raft_consensus.cc:3037] T 8db8c5faefbf454fa68d626a8e523654 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:58.159233 29484 raft_consensus.cc:504] T 8db8c5faefbf454fa68d626a8e523654 P 8148636210184b38b1fe012cd2d0eb29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:58.159981 29484 leader_election.cc:288] T 8db8c5faefbf454fa68d626a8e523654 P 8148636210184b38b1fe012cd2d0eb29 [CANDIDATE]: Term 1 election: Requested vote from peers 5da7dc5c70ff4f3fa6abf7cc75bb12f2 (127.27.196.65:42025), a3ecdc3dc21946cfb5f4a2e5ab0a75ba (127.27.196.67:36675)
I20240502 21:35:58.160710 29273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db8c5faefbf454fa68d626a8e523654" candidate_uuid: "8148636210184b38b1fe012cd2d0eb29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba"
I20240502 21:35:58.160684 29002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db8c5faefbf454fa68d626a8e523654" candidate_uuid: "8148636210184b38b1fe012cd2d0eb29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2"
I20240502 21:35:58.160929 29273 raft_consensus.cc:3037] T 8db8c5faefbf454fa68d626a8e523654 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:58.160959 29002 raft_consensus.cc:3037] T 8db8c5faefbf454fa68d626a8e523654 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:58.163009 29273 raft_consensus.cc:2454] T 8db8c5faefbf454fa68d626a8e523654 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8148636210184b38b1fe012cd2d0eb29 in term 1.
I20240502 21:35:58.163057 29002 raft_consensus.cc:2454] T 8db8c5faefbf454fa68d626a8e523654 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8148636210184b38b1fe012cd2d0eb29 in term 1.
I20240502 21:35:58.163607 29070 leader_election.cc:302] T 8db8c5faefbf454fa68d626a8e523654 P 8148636210184b38b1fe012cd2d0eb29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8148636210184b38b1fe012cd2d0eb29, a3ecdc3dc21946cfb5f4a2e5ab0a75ba; no voters: 
I20240502 21:35:58.164055 29484 raft_consensus.cc:2781] T 8db8c5faefbf454fa68d626a8e523654 P 8148636210184b38b1fe012cd2d0eb29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:58.164320 29484 raft_consensus.cc:686] T 8db8c5faefbf454fa68d626a8e523654 P 8148636210184b38b1fe012cd2d0eb29 [term 1 LEADER]: Becoming Leader. State: Replica: 8148636210184b38b1fe012cd2d0eb29, State: Running, Role: LEADER
I20240502 21:35:58.164804 29484 consensus_queue.cc:237] T 8db8c5faefbf454fa68d626a8e523654 P 8148636210184b38b1fe012cd2d0eb29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:58.168267 28854 catalog_manager.cc:5484] T 8db8c5faefbf454fa68d626a8e523654 P 8148636210184b38b1fe012cd2d0eb29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8148636210184b38b1fe012cd2d0eb29 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "8148636210184b38b1fe012cd2d0eb29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } health_report { overall_health: UNKNOWN } } }
I20240502 21:35:58.200125 29569 raft_consensus.cc:482] T bc25546ca0fe4e44a6edf394a6e13aa1 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:35:58.200362 29569 raft_consensus.cc:504] T bc25546ca0fe4e44a6edf394a6e13aa1 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:58.201329 29569 leader_election.cc:288] T bc25546ca0fe4e44a6edf394a6e13aa1 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5da7dc5c70ff4f3fa6abf7cc75bb12f2 (127.27.196.65:42025), 8148636210184b38b1fe012cd2d0eb29 (127.27.196.66:38577)
I20240502 21:35:58.202073 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc25546ca0fe4e44a6edf394a6e13aa1" candidate_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8148636210184b38b1fe012cd2d0eb29" is_pre_election: true
I20240502 21:35:58.202085 29002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc25546ca0fe4e44a6edf394a6e13aa1" candidate_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" is_pre_election: true
I20240502 21:35:58.202327 29002 raft_consensus.cc:2454] T bc25546ca0fe4e44a6edf394a6e13aa1 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3ecdc3dc21946cfb5f4a2e5ab0a75ba in term 0.
I20240502 21:35:58.202339 29136 raft_consensus.cc:2454] T bc25546ca0fe4e44a6edf394a6e13aa1 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3ecdc3dc21946cfb5f4a2e5ab0a75ba in term 0.
I20240502 21:35:58.202826 29206 leader_election.cc:302] T bc25546ca0fe4e44a6edf394a6e13aa1 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5da7dc5c70ff4f3fa6abf7cc75bb12f2, 8148636210184b38b1fe012cd2d0eb29, a3ecdc3dc21946cfb5f4a2e5ab0a75ba; no voters: 
I20240502 21:35:58.203209 29569 raft_consensus.cc:2781] T bc25546ca0fe4e44a6edf394a6e13aa1 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:35:58.203348 29569 raft_consensus.cc:482] T bc25546ca0fe4e44a6edf394a6e13aa1 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:35:58.203457 29569 raft_consensus.cc:3037] T bc25546ca0fe4e44a6edf394a6e13aa1 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:58.205742 29569 raft_consensus.cc:504] T bc25546ca0fe4e44a6edf394a6e13aa1 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:58.206516 29569 leader_election.cc:288] T bc25546ca0fe4e44a6edf394a6e13aa1 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [CANDIDATE]: Term 1 election: Requested vote from peers 5da7dc5c70ff4f3fa6abf7cc75bb12f2 (127.27.196.65:42025), 8148636210184b38b1fe012cd2d0eb29 (127.27.196.66:38577)
I20240502 21:35:58.207479 29002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc25546ca0fe4e44a6edf394a6e13aa1" candidate_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2"
I20240502 21:35:58.207424 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc25546ca0fe4e44a6edf394a6e13aa1" candidate_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8148636210184b38b1fe012cd2d0eb29"
I20240502 21:35:58.207700 29002 raft_consensus.cc:3037] T bc25546ca0fe4e44a6edf394a6e13aa1 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:58.207791 29136 raft_consensus.cc:3037] T bc25546ca0fe4e44a6edf394a6e13aa1 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:58.209893 29002 raft_consensus.cc:2454] T bc25546ca0fe4e44a6edf394a6e13aa1 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3ecdc3dc21946cfb5f4a2e5ab0a75ba in term 1.
I20240502 21:35:58.210361 29136 raft_consensus.cc:2454] T bc25546ca0fe4e44a6edf394a6e13aa1 P 8148636210184b38b1fe012cd2d0eb29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3ecdc3dc21946cfb5f4a2e5ab0a75ba in term 1.
I20240502 21:35:58.210692 29207 leader_election.cc:302] T bc25546ca0fe4e44a6edf394a6e13aa1 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5da7dc5c70ff4f3fa6abf7cc75bb12f2, a3ecdc3dc21946cfb5f4a2e5ab0a75ba; no voters: 
I20240502 21:35:58.211319 29569 raft_consensus.cc:2781] T bc25546ca0fe4e44a6edf394a6e13aa1 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:58.211588 29569 raft_consensus.cc:686] T bc25546ca0fe4e44a6edf394a6e13aa1 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 1 LEADER]: Becoming Leader. State: Replica: a3ecdc3dc21946cfb5f4a2e5ab0a75ba, State: Running, Role: LEADER
I20240502 21:35:58.211946 29569 consensus_queue.cc:237] T bc25546ca0fe4e44a6edf394a6e13aa1 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:58.215324 28854 catalog_manager.cc:5484] T bc25546ca0fe4e44a6edf394a6e13aa1 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba reported cstate change: term changed from 0 to 1, leader changed from <none> to a3ecdc3dc21946cfb5f4a2e5ab0a75ba (127.27.196.67). New cstate: current_term: 1 leader_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } health_report { overall_health: HEALTHY } } }
I20240502 21:35:58.562412 29504 consensus_queue.cc:1035] T ef659f7850914bb289fac74b6741544c P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:35:58.582145 29594 consensus_queue.cc:1035] T ef659f7850914bb289fac74b6741544c P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:35:58.643929 29569 consensus_queue.cc:1035] T bc25546ca0fe4e44a6edf394a6e13aa1 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:35:58.650017 29569 consensus_queue.cc:1035] T bc25546ca0fe4e44a6edf394a6e13aa1 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:35:58.654652 29484 consensus_queue.cc:1035] T 8db8c5faefbf454fa68d626a8e523654 P 8148636210184b38b1fe012cd2d0eb29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:35:58.661336 29498 consensus_queue.cc:1035] T 8db8c5faefbf454fa68d626a8e523654 P 8148636210184b38b1fe012cd2d0eb29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:35:58.685758 29589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:35:58.686317 29589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:35:58.699563 29589 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'

INSERT report
    rows total: 2000
    time total: 257.217 ms
  time per row: 0.128609 ms
I20240502 21:35:59.158507 28849 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59708:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240502 21:35:59.159471 28849 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 21:35:59.169145 28981 tablet_service.cc:1467] Processing CreateTablet for tablet 5a0481070ffc4f6b80bbb06667f361b2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=76d81c9b497a4dcaa39524b51725d5b4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:59.169145 29117 tablet_service.cc:1467] Processing CreateTablet for tablet 5a0481070ffc4f6b80bbb06667f361b2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=76d81c9b497a4dcaa39524b51725d5b4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:59.169680 28981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a0481070ffc4f6b80bbb06667f361b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:59.169680 29117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a0481070ffc4f6b80bbb06667f361b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:59.171617 29250 tablet_service.cc:1467] Processing CreateTablet for tablet 6bfe7bb2e16048c99f2cece8d89c3e77 (DEFAULT_TABLE table=rebalance_test_table_2 [id=76d81c9b497a4dcaa39524b51725d5b4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:59.171793 29251 tablet_service.cc:1467] Processing CreateTablet for tablet d75cc30bf1024f37bb53d03b898d70e7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=76d81c9b497a4dcaa39524b51725d5b4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:59.172118 29250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bfe7bb2e16048c99f2cece8d89c3e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:59.172482 29252 tablet_service.cc:1467] Processing CreateTablet for tablet 5a0481070ffc4f6b80bbb06667f361b2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=76d81c9b497a4dcaa39524b51725d5b4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:59.172482 29115 tablet_service.cc:1467] Processing CreateTablet for tablet d75cc30bf1024f37bb53d03b898d70e7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=76d81c9b497a4dcaa39524b51725d5b4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:59.173316 29252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a0481070ffc4f6b80bbb06667f361b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:59.173342 29115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d75cc30bf1024f37bb53d03b898d70e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:59.175314 29116 tablet_service.cc:1467] Processing CreateTablet for tablet 6bfe7bb2e16048c99f2cece8d89c3e77 (DEFAULT_TABLE table=rebalance_test_table_2 [id=76d81c9b497a4dcaa39524b51725d5b4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:59.175758 29116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bfe7bb2e16048c99f2cece8d89c3e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:59.176077 28979 tablet_service.cc:1467] Processing CreateTablet for tablet 6bfe7bb2e16048c99f2cece8d89c3e77 (DEFAULT_TABLE table=rebalance_test_table_2 [id=76d81c9b497a4dcaa39524b51725d5b4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:59.176779 28979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bfe7bb2e16048c99f2cece8d89c3e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:59.179363 29251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d75cc30bf1024f37bb53d03b898d70e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:59.183697 28980 tablet_service.cc:1467] Processing CreateTablet for tablet d75cc30bf1024f37bb53d03b898d70e7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=76d81c9b497a4dcaa39524b51725d5b4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:35:59.184298 28980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d75cc30bf1024f37bb53d03b898d70e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:35:59.195853 29654 tablet_bootstrap.cc:492] T 5a0481070ffc4f6b80bbb06667f361b2 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Bootstrap starting.
I20240502 21:35:59.198511 29654 tablet_bootstrap.cc:654] T 5a0481070ffc4f6b80bbb06667f361b2 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:59.201644 29653 tablet_bootstrap.cc:492] T 6bfe7bb2e16048c99f2cece8d89c3e77 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Bootstrap starting.
I20240502 21:35:59.202219 29652 tablet_bootstrap.cc:492] T d75cc30bf1024f37bb53d03b898d70e7 P 8148636210184b38b1fe012cd2d0eb29: Bootstrap starting.
I20240502 21:35:59.203675 29653 tablet_bootstrap.cc:654] T 6bfe7bb2e16048c99f2cece8d89c3e77 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:59.203747 29654 tablet_bootstrap.cc:492] T 5a0481070ffc4f6b80bbb06667f361b2 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: No bootstrap required, opened a new log
I20240502 21:35:59.203924 29654 ts_tablet_manager.cc:1367] T 5a0481070ffc4f6b80bbb06667f361b2 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Time spent bootstrapping tablet: real 0.008s	user 0.007s	sys 0.000s
I20240502 21:35:59.204048 29652 tablet_bootstrap.cc:654] T d75cc30bf1024f37bb53d03b898d70e7 P 8148636210184b38b1fe012cd2d0eb29: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:59.204927 29654 raft_consensus.cc:348] T 5a0481070ffc4f6b80bbb06667f361b2 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:59.205217 29654 raft_consensus.cc:374] T 5a0481070ffc4f6b80bbb06667f361b2 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:59.205320 29654 raft_consensus.cc:729] T 5a0481070ffc4f6b80bbb06667f361b2 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da7dc5c70ff4f3fa6abf7cc75bb12f2, State: Initialized, Role: FOLLOWER
I20240502 21:35:59.205688 29654 consensus_queue.cc:260] T 5a0481070ffc4f6b80bbb06667f361b2 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:59.206075 29653 tablet_bootstrap.cc:492] T 6bfe7bb2e16048c99f2cece8d89c3e77 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: No bootstrap required, opened a new log
I20240502 21:35:59.206223 29653 ts_tablet_manager.cc:1367] T 6bfe7bb2e16048c99f2cece8d89c3e77 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.003s
I20240502 21:35:59.206387 29652 tablet_bootstrap.cc:492] T d75cc30bf1024f37bb53d03b898d70e7 P 8148636210184b38b1fe012cd2d0eb29: No bootstrap required, opened a new log
I20240502 21:35:59.206533 29652 ts_tablet_manager.cc:1367] T d75cc30bf1024f37bb53d03b898d70e7 P 8148636210184b38b1fe012cd2d0eb29: Time spent bootstrapping tablet: real 0.004s	user 0.004s	sys 0.000s
I20240502 21:35:59.207227 29654 ts_tablet_manager.cc:1398] T 5a0481070ffc4f6b80bbb06667f361b2 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 21:35:59.207067 29653 raft_consensus.cc:348] T 6bfe7bb2e16048c99f2cece8d89c3e77 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:59.207332 29653 raft_consensus.cc:374] T 6bfe7bb2e16048c99f2cece8d89c3e77 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:59.207438 29653 raft_consensus.cc:729] T 6bfe7bb2e16048c99f2cece8d89c3e77 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3ecdc3dc21946cfb5f4a2e5ab0a75ba, State: Initialized, Role: FOLLOWER
I20240502 21:35:59.207564 29654 tablet_bootstrap.cc:492] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Bootstrap starting.
I20240502 21:35:59.207641 29652 raft_consensus.cc:348] T d75cc30bf1024f37bb53d03b898d70e7 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:59.207844 29652 raft_consensus.cc:374] T d75cc30bf1024f37bb53d03b898d70e7 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:59.207715 29653 consensus_queue.cc:260] T 6bfe7bb2e16048c99f2cece8d89c3e77 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:59.207954 29652 raft_consensus.cc:729] T d75cc30bf1024f37bb53d03b898d70e7 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8148636210184b38b1fe012cd2d0eb29, State: Initialized, Role: FOLLOWER
I20240502 21:35:59.208323 29653 ts_tablet_manager.cc:1398] T 6bfe7bb2e16048c99f2cece8d89c3e77 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 21:35:59.208266 29652 consensus_queue.cc:260] T d75cc30bf1024f37bb53d03b898d70e7 P 8148636210184b38b1fe012cd2d0eb29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:59.208628 29653 tablet_bootstrap.cc:492] T 5a0481070ffc4f6b80bbb06667f361b2 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Bootstrap starting.
I20240502 21:35:59.209240 29654 tablet_bootstrap.cc:654] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:59.210474 29653 tablet_bootstrap.cc:654] T 5a0481070ffc4f6b80bbb06667f361b2 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:59.210487 29652 ts_tablet_manager.cc:1398] T d75cc30bf1024f37bb53d03b898d70e7 P 8148636210184b38b1fe012cd2d0eb29: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 21:35:59.211016 29652 tablet_bootstrap.cc:492] T 5a0481070ffc4f6b80bbb06667f361b2 P 8148636210184b38b1fe012cd2d0eb29: Bootstrap starting.
I20240502 21:35:59.213363 29652 tablet_bootstrap.cc:654] T 5a0481070ffc4f6b80bbb06667f361b2 P 8148636210184b38b1fe012cd2d0eb29: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:59.218904 29654 tablet_bootstrap.cc:492] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: No bootstrap required, opened a new log
I20240502 21:35:59.219235 29654 ts_tablet_manager.cc:1367] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Time spent bootstrapping tablet: real 0.012s	user 0.005s	sys 0.005s
I20240502 21:35:59.220551 29654 raft_consensus.cc:348] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:59.220871 29654 raft_consensus.cc:374] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:59.220960 29654 raft_consensus.cc:729] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da7dc5c70ff4f3fa6abf7cc75bb12f2, State: Initialized, Role: FOLLOWER
I20240502 21:35:59.221351 29654 consensus_queue.cc:260] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:59.222455 29654 ts_tablet_manager.cc:1398] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 21:35:59.222918 29654 tablet_bootstrap.cc:492] T d75cc30bf1024f37bb53d03b898d70e7 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Bootstrap starting.
I20240502 21:35:59.226435 29654 tablet_bootstrap.cc:654] T d75cc30bf1024f37bb53d03b898d70e7 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:59.226573 29652 tablet_bootstrap.cc:492] T 5a0481070ffc4f6b80bbb06667f361b2 P 8148636210184b38b1fe012cd2d0eb29: No bootstrap required, opened a new log
I20240502 21:35:59.226794 29652 ts_tablet_manager.cc:1367] T 5a0481070ffc4f6b80bbb06667f361b2 P 8148636210184b38b1fe012cd2d0eb29: Time spent bootstrapping tablet: real 0.016s	user 0.003s	sys 0.007s
I20240502 21:35:59.227811 29652 raft_consensus.cc:348] T 5a0481070ffc4f6b80bbb06667f361b2 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:59.228080 29652 raft_consensus.cc:374] T 5a0481070ffc4f6b80bbb06667f361b2 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:59.228163 29652 raft_consensus.cc:729] T 5a0481070ffc4f6b80bbb06667f361b2 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8148636210184b38b1fe012cd2d0eb29, State: Initialized, Role: FOLLOWER
I20240502 21:35:59.228509 29652 consensus_queue.cc:260] T 5a0481070ffc4f6b80bbb06667f361b2 P 8148636210184b38b1fe012cd2d0eb29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:59.232471 29652 ts_tablet_manager.cc:1398] T 5a0481070ffc4f6b80bbb06667f361b2 P 8148636210184b38b1fe012cd2d0eb29: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20240502 21:35:59.235756 29654 tablet_bootstrap.cc:492] T d75cc30bf1024f37bb53d03b898d70e7 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: No bootstrap required, opened a new log
I20240502 21:35:59.235985 29654 ts_tablet_manager.cc:1367] T d75cc30bf1024f37bb53d03b898d70e7 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Time spent bootstrapping tablet: real 0.013s	user 0.010s	sys 0.000s
I20240502 21:35:59.236387 29652 tablet_bootstrap.cc:492] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 8148636210184b38b1fe012cd2d0eb29: Bootstrap starting.
I20240502 21:35:59.237090 29654 raft_consensus.cc:348] T d75cc30bf1024f37bb53d03b898d70e7 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:59.237404 29654 raft_consensus.cc:374] T d75cc30bf1024f37bb53d03b898d70e7 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:59.237529 29654 raft_consensus.cc:729] T d75cc30bf1024f37bb53d03b898d70e7 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da7dc5c70ff4f3fa6abf7cc75bb12f2, State: Initialized, Role: FOLLOWER
I20240502 21:35:59.237949 29654 consensus_queue.cc:260] T d75cc30bf1024f37bb53d03b898d70e7 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:59.238402 29652 tablet_bootstrap.cc:654] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 8148636210184b38b1fe012cd2d0eb29: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:59.240309 29654 ts_tablet_manager.cc:1398] T d75cc30bf1024f37bb53d03b898d70e7 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 21:35:59.246212 29653 tablet_bootstrap.cc:492] T 5a0481070ffc4f6b80bbb06667f361b2 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: No bootstrap required, opened a new log
I20240502 21:35:59.246510 29652 tablet_bootstrap.cc:492] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 8148636210184b38b1fe012cd2d0eb29: No bootstrap required, opened a new log
I20240502 21:35:59.246511 29653 ts_tablet_manager.cc:1367] T 5a0481070ffc4f6b80bbb06667f361b2 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Time spent bootstrapping tablet: real 0.038s	user 0.012s	sys 0.024s
I20240502 21:35:59.246706 29652 ts_tablet_manager.cc:1367] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 8148636210184b38b1fe012cd2d0eb29: Time spent bootstrapping tablet: real 0.010s	user 0.004s	sys 0.004s
I20240502 21:35:59.247890 29653 raft_consensus.cc:348] T 5a0481070ffc4f6b80bbb06667f361b2 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:59.247898 29652 raft_consensus.cc:348] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:59.248281 29652 raft_consensus.cc:374] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:59.248315 29653 raft_consensus.cc:374] T 5a0481070ffc4f6b80bbb06667f361b2 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:59.248410 29653 raft_consensus.cc:729] T 5a0481070ffc4f6b80bbb06667f361b2 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3ecdc3dc21946cfb5f4a2e5ab0a75ba, State: Initialized, Role: FOLLOWER
I20240502 21:35:59.248432 29652 raft_consensus.cc:729] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8148636210184b38b1fe012cd2d0eb29, State: Initialized, Role: FOLLOWER
I20240502 21:35:59.248744 29652 consensus_queue.cc:260] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 8148636210184b38b1fe012cd2d0eb29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:59.248840 29653 consensus_queue.cc:260] T 5a0481070ffc4f6b80bbb06667f361b2 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:59.249636 29652 ts_tablet_manager.cc:1398] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 8148636210184b38b1fe012cd2d0eb29: Time spent starting tablet: real 0.003s	user 0.004s	sys 0.000s
I20240502 21:35:59.249840 29653 ts_tablet_manager.cc:1398] T 5a0481070ffc4f6b80bbb06667f361b2 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Time spent starting tablet: real 0.003s	user 0.004s	sys 0.000s
I20240502 21:35:59.250247 29653 tablet_bootstrap.cc:492] T d75cc30bf1024f37bb53d03b898d70e7 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Bootstrap starting.
I20240502 21:35:59.252264 29653 tablet_bootstrap.cc:654] T d75cc30bf1024f37bb53d03b898d70e7 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Neither blocks nor log segments found. Creating new log.
I20240502 21:35:59.256589 29653 tablet_bootstrap.cc:492] T d75cc30bf1024f37bb53d03b898d70e7 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: No bootstrap required, opened a new log
I20240502 21:35:59.256773 29653 ts_tablet_manager.cc:1367] T d75cc30bf1024f37bb53d03b898d70e7 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.004s
I20240502 21:35:59.257565 29653 raft_consensus.cc:348] T d75cc30bf1024f37bb53d03b898d70e7 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:59.257776 29653 raft_consensus.cc:374] T d75cc30bf1024f37bb53d03b898d70e7 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:35:59.257882 29653 raft_consensus.cc:729] T d75cc30bf1024f37bb53d03b898d70e7 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3ecdc3dc21946cfb5f4a2e5ab0a75ba, State: Initialized, Role: FOLLOWER
I20240502 21:35:59.258157 29653 consensus_queue.cc:260] T d75cc30bf1024f37bb53d03b898d70e7 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:59.258702 29653 ts_tablet_manager.cc:1398] T d75cc30bf1024f37bb53d03b898d70e7 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 21:35:59.302825 29488 raft_consensus.cc:482] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:35:59.303081 29488 raft_consensus.cc:504] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:59.303920 29488 leader_election.cc:288] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8148636210184b38b1fe012cd2d0eb29 (127.27.196.66:38577), a3ecdc3dc21946cfb5f4a2e5ab0a75ba (127.27.196.67:36675)
I20240502 21:35:59.304761 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bfe7bb2e16048c99f2cece8d89c3e77" candidate_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8148636210184b38b1fe012cd2d0eb29" is_pre_election: true
I20240502 21:35:59.305040 29136 raft_consensus.cc:2454] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5da7dc5c70ff4f3fa6abf7cc75bb12f2 in term 0.
I20240502 21:35:59.305032 29272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bfe7bb2e16048c99f2cece8d89c3e77" candidate_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" is_pre_election: true
I20240502 21:35:59.305521 29272 raft_consensus.cc:2454] T 6bfe7bb2e16048c99f2cece8d89c3e77 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5da7dc5c70ff4f3fa6abf7cc75bb12f2 in term 0.
I20240502 21:35:59.305644 28933 leader_election.cc:302] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5da7dc5c70ff4f3fa6abf7cc75bb12f2, 8148636210184b38b1fe012cd2d0eb29; no voters: 
I20240502 21:35:59.305994 29488 raft_consensus.cc:2781] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:35:59.306154 29488 raft_consensus.cc:482] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:35:59.306262 29488 raft_consensus.cc:3037] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:59.308687 29488 raft_consensus.cc:504] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:59.309747 29488 leader_election.cc:288] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [CANDIDATE]: Term 1 election: Requested vote from peers 8148636210184b38b1fe012cd2d0eb29 (127.27.196.66:38577), a3ecdc3dc21946cfb5f4a2e5ab0a75ba (127.27.196.67:36675)
I20240502 21:35:59.310365 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bfe7bb2e16048c99f2cece8d89c3e77" candidate_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8148636210184b38b1fe012cd2d0eb29"
I20240502 21:35:59.310433 29272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bfe7bb2e16048c99f2cece8d89c3e77" candidate_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba"
I20240502 21:35:59.310711 29272 raft_consensus.cc:3037] T 6bfe7bb2e16048c99f2cece8d89c3e77 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:59.310701 29136 raft_consensus.cc:3037] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:59.313393 29272 raft_consensus.cc:2454] T 6bfe7bb2e16048c99f2cece8d89c3e77 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5da7dc5c70ff4f3fa6abf7cc75bb12f2 in term 1.
I20240502 21:35:59.313937 28933 leader_election.cc:302] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5da7dc5c70ff4f3fa6abf7cc75bb12f2, a3ecdc3dc21946cfb5f4a2e5ab0a75ba; no voters: 
I20240502 21:35:59.314050 29136 raft_consensus.cc:2454] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 8148636210184b38b1fe012cd2d0eb29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5da7dc5c70ff4f3fa6abf7cc75bb12f2 in term 1.
I20240502 21:35:59.314289 29488 raft_consensus.cc:2781] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:59.314726 29488 raft_consensus.cc:686] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 1 LEADER]: Becoming Leader. State: Replica: 5da7dc5c70ff4f3fa6abf7cc75bb12f2, State: Running, Role: LEADER
I20240502 21:35:59.315034 29488 consensus_queue.cc:237] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:59.318388 28854 catalog_manager.cc:5484] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5da7dc5c70ff4f3fa6abf7cc75bb12f2 (127.27.196.65). New cstate: current_term: 1 leader_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } health_report { overall_health: UNKNOWN } } }
I20240502 21:35:59.336714 29645 raft_consensus.cc:482] T 5a0481070ffc4f6b80bbb06667f361b2 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:35:59.336956 29645 raft_consensus.cc:504] T 5a0481070ffc4f6b80bbb06667f361b2 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:59.337771 29645 leader_election.cc:288] T 5a0481070ffc4f6b80bbb06667f361b2 P 8148636210184b38b1fe012cd2d0eb29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5da7dc5c70ff4f3fa6abf7cc75bb12f2 (127.27.196.65:42025), a3ecdc3dc21946cfb5f4a2e5ab0a75ba (127.27.196.67:36675)
I20240502 21:35:59.338443 29272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a0481070ffc4f6b80bbb06667f361b2" candidate_uuid: "8148636210184b38b1fe012cd2d0eb29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" is_pre_election: true
I20240502 21:35:59.338467 29002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a0481070ffc4f6b80bbb06667f361b2" candidate_uuid: "8148636210184b38b1fe012cd2d0eb29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" is_pre_election: true
I20240502 21:35:59.338727 29272 raft_consensus.cc:2454] T 5a0481070ffc4f6b80bbb06667f361b2 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8148636210184b38b1fe012cd2d0eb29 in term 0.
I20240502 21:35:59.338734 29002 raft_consensus.cc:2454] T 5a0481070ffc4f6b80bbb06667f361b2 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8148636210184b38b1fe012cd2d0eb29 in term 0.
I20240502 21:35:59.339299 29070 leader_election.cc:302] T 5a0481070ffc4f6b80bbb06667f361b2 P 8148636210184b38b1fe012cd2d0eb29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8148636210184b38b1fe012cd2d0eb29, a3ecdc3dc21946cfb5f4a2e5ab0a75ba; no voters: 
I20240502 21:35:59.339643 29645 raft_consensus.cc:2781] T 5a0481070ffc4f6b80bbb06667f361b2 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:35:59.339757 29645 raft_consensus.cc:482] T 5a0481070ffc4f6b80bbb06667f361b2 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:35:59.339851 29645 raft_consensus.cc:3037] T 5a0481070ffc4f6b80bbb06667f361b2 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:59.341975 29645 raft_consensus.cc:504] T 5a0481070ffc4f6b80bbb06667f361b2 P 8148636210184b38b1fe012cd2d0eb29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:59.342746 29645 leader_election.cc:288] T 5a0481070ffc4f6b80bbb06667f361b2 P 8148636210184b38b1fe012cd2d0eb29 [CANDIDATE]: Term 1 election: Requested vote from peers 5da7dc5c70ff4f3fa6abf7cc75bb12f2 (127.27.196.65:42025), a3ecdc3dc21946cfb5f4a2e5ab0a75ba (127.27.196.67:36675)
I20240502 21:35:59.343389 29002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a0481070ffc4f6b80bbb06667f361b2" candidate_uuid: "8148636210184b38b1fe012cd2d0eb29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2"
I20240502 21:35:59.343637 29002 raft_consensus.cc:3037] T 5a0481070ffc4f6b80bbb06667f361b2 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:59.343538 29272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a0481070ffc4f6b80bbb06667f361b2" candidate_uuid: "8148636210184b38b1fe012cd2d0eb29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba"
I20240502 21:35:59.344122 29272 raft_consensus.cc:3037] T 5a0481070ffc4f6b80bbb06667f361b2 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:59.346293 29002 raft_consensus.cc:2454] T 5a0481070ffc4f6b80bbb06667f361b2 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8148636210184b38b1fe012cd2d0eb29 in term 1.
I20240502 21:35:59.346987 29071 leader_election.cc:302] T 5a0481070ffc4f6b80bbb06667f361b2 P 8148636210184b38b1fe012cd2d0eb29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5da7dc5c70ff4f3fa6abf7cc75bb12f2, 8148636210184b38b1fe012cd2d0eb29; no voters: 
I20240502 21:35:59.347307 29272 raft_consensus.cc:2454] T 5a0481070ffc4f6b80bbb06667f361b2 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8148636210184b38b1fe012cd2d0eb29 in term 1.
I20240502 21:35:59.347682 29645 raft_consensus.cc:2781] T 5a0481070ffc4f6b80bbb06667f361b2 P 8148636210184b38b1fe012cd2d0eb29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:59.347873 29645 raft_consensus.cc:686] T 5a0481070ffc4f6b80bbb06667f361b2 P 8148636210184b38b1fe012cd2d0eb29 [term 1 LEADER]: Becoming Leader. State: Replica: 8148636210184b38b1fe012cd2d0eb29, State: Running, Role: LEADER
I20240502 21:35:59.348650 29645 consensus_queue.cc:237] T 5a0481070ffc4f6b80bbb06667f361b2 P 8148636210184b38b1fe012cd2d0eb29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:59.355962 28854 catalog_manager.cc:5484] T 5a0481070ffc4f6b80bbb06667f361b2 P 8148636210184b38b1fe012cd2d0eb29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8148636210184b38b1fe012cd2d0eb29 (127.27.196.66). New cstate: current_term: 1 leader_uuid: "8148636210184b38b1fe012cd2d0eb29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } health_report { overall_health: UNKNOWN } } }
I20240502 21:35:59.401355 29569 raft_consensus.cc:482] T d75cc30bf1024f37bb53d03b898d70e7 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:35:59.401605 29569 raft_consensus.cc:504] T d75cc30bf1024f37bb53d03b898d70e7 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:59.402336 29569 leader_election.cc:288] T d75cc30bf1024f37bb53d03b898d70e7 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8148636210184b38b1fe012cd2d0eb29 (127.27.196.66:38577), 5da7dc5c70ff4f3fa6abf7cc75bb12f2 (127.27.196.65:42025)
I20240502 21:35:59.403153 29002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75cc30bf1024f37bb53d03b898d70e7" candidate_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" is_pre_election: true
I20240502 21:35:59.403103 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75cc30bf1024f37bb53d03b898d70e7" candidate_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8148636210184b38b1fe012cd2d0eb29" is_pre_election: true
I20240502 21:35:59.403422 29002 raft_consensus.cc:2454] T d75cc30bf1024f37bb53d03b898d70e7 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3ecdc3dc21946cfb5f4a2e5ab0a75ba in term 0.
I20240502 21:35:59.403443 29136 raft_consensus.cc:2454] T d75cc30bf1024f37bb53d03b898d70e7 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3ecdc3dc21946cfb5f4a2e5ab0a75ba in term 0.
I20240502 21:35:59.403982 29206 leader_election.cc:302] T d75cc30bf1024f37bb53d03b898d70e7 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8148636210184b38b1fe012cd2d0eb29, a3ecdc3dc21946cfb5f4a2e5ab0a75ba; no voters: 
I20240502 21:35:59.404402 29569 raft_consensus.cc:2781] T d75cc30bf1024f37bb53d03b898d70e7 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:35:59.404502 29569 raft_consensus.cc:482] T d75cc30bf1024f37bb53d03b898d70e7 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:35:59.404619 29569 raft_consensus.cc:3037] T d75cc30bf1024f37bb53d03b898d70e7 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:59.406721 29569 raft_consensus.cc:504] T d75cc30bf1024f37bb53d03b898d70e7 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:59.407435 29569 leader_election.cc:288] T d75cc30bf1024f37bb53d03b898d70e7 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [CANDIDATE]: Term 1 election: Requested vote from peers 8148636210184b38b1fe012cd2d0eb29 (127.27.196.66:38577), 5da7dc5c70ff4f3fa6abf7cc75bb12f2 (127.27.196.65:42025)
I20240502 21:35:59.408138 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75cc30bf1024f37bb53d03b898d70e7" candidate_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8148636210184b38b1fe012cd2d0eb29"
I20240502 21:35:59.408396 29136 raft_consensus.cc:3037] T d75cc30bf1024f37bb53d03b898d70e7 P 8148636210184b38b1fe012cd2d0eb29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:59.408392 29002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75cc30bf1024f37bb53d03b898d70e7" candidate_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2"
I20240502 21:35:59.408751 29002 raft_consensus.cc:3037] T d75cc30bf1024f37bb53d03b898d70e7 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:35:59.411168 29136 raft_consensus.cc:2454] T d75cc30bf1024f37bb53d03b898d70e7 P 8148636210184b38b1fe012cd2d0eb29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3ecdc3dc21946cfb5f4a2e5ab0a75ba in term 1.
I20240502 21:35:59.411190 29002 raft_consensus.cc:2454] T d75cc30bf1024f37bb53d03b898d70e7 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3ecdc3dc21946cfb5f4a2e5ab0a75ba in term 1.
I20240502 21:35:59.411893 29207 leader_election.cc:302] T d75cc30bf1024f37bb53d03b898d70e7 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5da7dc5c70ff4f3fa6abf7cc75bb12f2, 8148636210184b38b1fe012cd2d0eb29, a3ecdc3dc21946cfb5f4a2e5ab0a75ba; no voters: 
I20240502 21:35:59.412386 29569 raft_consensus.cc:2781] T d75cc30bf1024f37bb53d03b898d70e7 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:35:59.412614 29569 raft_consensus.cc:686] T d75cc30bf1024f37bb53d03b898d70e7 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [term 1 LEADER]: Becoming Leader. State: Replica: a3ecdc3dc21946cfb5f4a2e5ab0a75ba, State: Running, Role: LEADER
I20240502 21:35:59.412977 29569 consensus_queue.cc:237] T d75cc30bf1024f37bb53d03b898d70e7 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } }
I20240502 21:35:59.416464 28854 catalog_manager.cc:5484] T d75cc30bf1024f37bb53d03b898d70e7 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba reported cstate change: term changed from 0 to 1, leader changed from <none> to a3ecdc3dc21946cfb5f4a2e5ab0a75ba (127.27.196.67). New cstate: current_term: 1 leader_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 } health_report { overall_health: HEALTHY } } }
I20240502 21:35:59.738437 29645 consensus_queue.cc:1035] T 5a0481070ffc4f6b80bbb06667f361b2 P 8148636210184b38b1fe012cd2d0eb29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:35:59.746500 29645 consensus_queue.cc:1035] T 5a0481070ffc4f6b80bbb06667f361b2 P 8148636210184b38b1fe012cd2d0eb29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:35:59.752068 29488 consensus_queue.cc:1035] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3ecdc3dc21946cfb5f4a2e5ab0a75ba" member_type: VOTER last_known_addr { host: "127.27.196.67" port: 36675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:35:59.765661 29504 consensus_queue.cc:1035] T 6bfe7bb2e16048c99f2cece8d89c3e77 P 5da7dc5c70ff4f3fa6abf7cc75bb12f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:35:59.845134 29569 consensus_queue.cc:1035] T d75cc30bf1024f37bb53d03b898d70e7 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "8148636210184b38b1fe012cd2d0eb29" member_type: VOTER last_known_addr { host: "127.27.196.66" port: 38577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:35:59.852917 29530 consensus_queue.cc:1035] T d75cc30bf1024f37bb53d03b898d70e7 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5da7dc5c70ff4f3fa6abf7cc75bb12f2" member_type: VOTER last_known_addr { host: "127.27.196.65" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:35:59.869185 29667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:35:59.869634 29667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:35:59.877897 29667 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'

INSERT report
    rows total: 2000
    time total: 254.191 ms
  time per row: 0.127096 ms
I20240502 21:36:00.328884 28433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
/tmp/dist-test-taskBPlvDh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.196.68:39161
--local_ip_for_outbound_sockets=127.27.196.68
--tserver_master_addrs=127.27.196.126:46355
--webserver_port=38177
--webserver_interface=127.27.196.68
--builtin_ntp_servers=127.27.196.84:33157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:36:00.754261 29727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:36:00.754623 29727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:36:00.754791 29727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:36:00.763303 29727 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:36:00.763486 29727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:36:00.763646 29727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.196.68
I20240502 21:36:00.773363 29727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.196.84:33157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.196.68:39161
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.27.196.68
--webserver_port=38177
--tserver_master_addrs=127.27.196.126:46355
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.196.68
--log_dir=/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:36:00.774796 29727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:36:00.776760 29727 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:36:00.791174 29732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:36:00.791568 29735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:36:00.792117 29733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:36:00.792682 29727 server_base.cc:1034] running on GCE node
I20240502 21:36:00.793373 29727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:36:00.794474 29727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:36:00.795786 29727 hybrid_clock.cc:648] HybridClock initialized: now 1714685760795711 us; error 37 us; skew 500 ppm
I20240502 21:36:00.796286 29727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:36:00.799552 29727 webserver.cc:460] Webserver started at http://127.27.196.68:38177/ using document root <none> and password file <none>
I20240502 21:36:00.800580 29727 fs_manager.cc:359] Metadata directory not provided
I20240502 21:36:00.800732 29727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:36:00.808140 29727 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.007s	sys 0.000s
I20240502 21:36:00.811446 29741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:36:00.813390 29727 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
I20240502 21:36:00.813561 29727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "05a47f25c377450b930558c362256996"
format_stamp: "Formatted at 2024-05-02 21:35:52 on dist-test-slave-3bs1"
I20240502 21:36:00.814522 29727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:36:00.836177 29727 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 21:36:00.837213 29727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:36:00.837525 29727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:36:00.838630 29727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:36:00.840482 29727 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:36:00.840560 29727 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:36:00.840669 29727 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:36:00.840734 29727 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 21:36:00.878154 29727 rpc_server.cc:307] RPC server started. Bound to: 127.27.196.68:39161
I20240502 21:36:00.878202 29854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.196.68:39161 every 8 connection(s)
I20240502 21:36:00.880131 29727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBPlvDh/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714685119845993-28433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:36:00.889799 28433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu as pid 29727
I20240502 21:36:00.896823 29855 heartbeater.cc:347] Connected to a master server at 127.27.196.126:46355
I20240502 21:36:00.897233 29855 heartbeater.cc:464] Registering TS with master...
I20240502 21:36:00.898387 29855 heartbeater.cc:511] Master 127.27.196.126:46355 requested a full tablet report, sending...
I20240502 21:36:00.901114 28854 ts_manager.cc:194] Re-registered known tserver with Master: 05a47f25c377450b930558c362256996 (127.27.196.68:39161)
I20240502 21:36:00.902061 28854 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.196.68:59355
I20240502 21:36:01.473860 28981 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:36:01.475427 29789 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:36:01.486711 29115 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:36:01.490921 29250 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:36:01.707448 29789 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:36:01.715271 29115 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:36:01.716331 28981 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:36:01.722775 29250 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:36:01.905617 29855 heartbeater.cc:503] Master 127.27.196.126:46355 was elected leader, sending a full tablet report...
I20240502 21:36:01.936249 29789 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:36:01.945173 29115 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:36:01.951078 28981 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:36:01.960673 29250 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:36:02.626273 28854 ts_manager.cc:292] Set tserver state for 05a47f25c377450b930558c362256996 to MAINTENANCE_MODE
I20240502 21:36:03.355151 29789 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:36:03.358989 28981 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:36:03.361541 29115 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:36:03.361641 29250 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:36:03.559309 29789 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:36:03.562909 29115 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:36:03.568024 28981 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:36:03.571491 29250 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:36:03.777766 29250 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:36:03.780740 28981 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:36:03.781651 29789 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:36:03.784399 29115 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:36:03.975811 29789 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:36:03.987074 29115 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:36:03.987176 29250 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:36:03.988595 28981 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:36:04.180613 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 28916
I20240502 21:36:04.195005 29043 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:36:04.329666 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 28916
W20240502 21:36:04.362334 29071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.196.65:42025 (error 108)
I20240502 21:36:04.363029 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 29053
W20240502 21:36:04.365516 29207 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.196.65:42025: connect: Connection refused (error 111)
W20240502 21:36:04.365708 29071 consensus_peers.cc:485] T b44bc823d03c42b690b1e3fef7371e7b P 8148636210184b38b1fe012cd2d0eb29 -> Peer 5da7dc5c70ff4f3fa6abf7cc75bb12f2 (127.27.196.65:42025): Couldn't send request to peer 5da7dc5c70ff4f3fa6abf7cc75bb12f2. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:42025: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:36:04.365936 29071 consensus_peers.cc:485] T 5a0481070ffc4f6b80bbb06667f361b2 P 8148636210184b38b1fe012cd2d0eb29 -> Peer 5da7dc5c70ff4f3fa6abf7cc75bb12f2 (127.27.196.65:42025): Couldn't send request to peer 5da7dc5c70ff4f3fa6abf7cc75bb12f2. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:42025: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:36:04.366123 29071 consensus_peers.cc:485] T 8db8c5faefbf454fa68d626a8e523654 P 8148636210184b38b1fe012cd2d0eb29 -> Peer 5da7dc5c70ff4f3fa6abf7cc75bb12f2 (127.27.196.65:42025): Couldn't send request to peer 5da7dc5c70ff4f3fa6abf7cc75bb12f2. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:42025: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:36:04.367574 29207 consensus_peers.cc:485] T 87f738af6dec4151b08895d4f753fc5e P a3ecdc3dc21946cfb5f4a2e5ab0a75ba -> Peer 5da7dc5c70ff4f3fa6abf7cc75bb12f2 (127.27.196.65:42025): Couldn't send request to peer 5da7dc5c70ff4f3fa6abf7cc75bb12f2. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.65:42025: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:36:04.371944 29179 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:36:04.516808 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 29053
I20240502 21:36:04.549805 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 29189
W20240502 21:36:04.550788 29206 consensus_peers.cc:485] T 87f738af6dec4151b08895d4f753fc5e P a3ecdc3dc21946cfb5f4a2e5ab0a75ba -> Peer 8148636210184b38b1fe012cd2d0eb29 (127.27.196.66:38577): Couldn't send request to peer 8148636210184b38b1fe012cd2d0eb29. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.66:38577: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:36:04.551002 29206 consensus_peers.cc:485] T bc25546ca0fe4e44a6edf394a6e13aa1 P a3ecdc3dc21946cfb5f4a2e5ab0a75ba -> Peer 8148636210184b38b1fe012cd2d0eb29 (127.27.196.66:38577): Couldn't send request to peer 8148636210184b38b1fe012cd2d0eb29. Status: Network error: Client connection negotiation failed: client connection to 127.27.196.66:38577: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:36:04.558452 29315 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:36:04.695959 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 29189
I20240502 21:36:04.723942 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 29727
I20240502 21:36:04.732873 29849 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:36:04.839506 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 29727
I20240502 21:36:04.864882 28433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu pid 28823
I20240502 21:36:04.866154 28887 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:36:05.026671 28433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBPlvDh/build/asan/bin/kudu with pid 28823
2024-05-02T21:36:05Z chronyd exiting
[       OK ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx1_rf3_t3_l_3_1 (15268 ms)
[----------] 2 tests from LocationAwareRebalancingParamTest (39300 ms total)

[----------] Global test environment tear-down
[==========] 7 tests from 6 test suites ran. (645217 ms total)
[  PASSED  ] 6 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0, where GetParam() = 4-byte object <00-00 00-00>

 1 FAILED TEST